SSL renegotiation with full duplex socket communication

SSL renegotiation with full duplex socket communication

I have a very simple client-server with one blocking socket doing
full-duplex communication. I've enabled SSL/TLS to the application. The
model is that of a typical producer-consumer. The client produces the
data, sends it to the server and the server processes them. The only catch
is that, once in a while the server sends data back to the client which
the client handles accordingly. Below is a very simple pseudo code of the
application:
1 Client:
2 -------
3 while (true)
4 {
5 if (poll(pollin, timeout=0) || 0 < SSL_pending(ssl))
6 {
7 SSL_read();
8 // Handle WANT_READ or WANT_WRITE appropriately.
9 // If no error, handle the received control message.
10 }
11 // produce data.
12 while (!poll(pollout))
13 ; // Wait until the pipe is ready for a send().
14 SSL_write();
15 // Handle WANT_READ or WANT_WRITE appropriately.
16 if (time to renegotiate)
17 SSL_renegotiate(ssl);
18 }
19
20 Server:
21 -------
22 while (true)
23 {
24 if (poll(pollin, timeout=1s) || 0 < SSL_pending(ssl))
25 {
26 SSL_read();
27 // Handle WANT_READ or WANT_WRITE appropriately.
28 // If no error, consume data.
29 }
30 if (control message needs to be sent)
31 {
32 while (!poll(pollout))
33 ; // Wait until the pipe is ready for a send().
34 SSL_write();
35 // Handle WANT_READ or WANT_WRITE appropriately.
36 }
37 }
The trouble happens when, for testing purposes, I force SSL renegotiation
(lines 16-17). The session starts nice and easy, but after a while, I get
the following errors:
Client:
-------
error:140940F5:SSL routines:SSL3_READ_BYTES:unexpected record
Server:
-------
error:140943F2:SSL routines:SSL3_READ_BYTES:sslv3 alert unexpected message
Turns out, around the same time that the client initiates a renegotiation
(line 14), the server ends up sending application data to the client (line
34). The client as part of the renegotiation process receives this
application data and bombs with a "unexpected record" error. Similarly,
when the server does the subsequent receive (line 26), it ends up
receiving a renegotiation data when it was expecting application data.
What am I doing wrong? How should I handle/test SSL renegotiations with a
full-duplex channel. Note that, there are no threads involved. It's a
simple single threaded model with reads/writes happening on either end of
the socket.