id summary reporter owner description type status priority milestone component version resolution keywords cc uname nginx_version 413 Extra roundtrip during SSL handshake with long certificate chains Jiri Horky "Hi, during some performance testing I noticed that there is an extra RTT involved during SSL handshake when using big-enough certificate file (4096-bit self-signed certificate is big enough to trigger the error). I have located the problem in the size of openSSL's write buffers: DEFAULT_BUFFER_SIZE=4096 in crypto/bio/bf_buff.c which causes that the certificate does not get to the client during one write(2) to the socket, as only as many bytes as the buffer size are flushed in one moment. The rest of bytes are cached until openSSL does not have anything else to send or decides to flush, which, unfortunatelly waits for some ACK from client. Here is an exterm of the communication from tcpdump (run on the client), the RTT is 100ms, this shows simple openssl s_client -connect operation (just SSL handshake): 1. 06:06:28.837676 IP client.42233 > server.com.https: Flags [S], seq 2917118389, win 2920, options [mss 1460,sackOK,TS val 32817878 ecr 0,nop,wscale 13], length 0 2. 06:06:28.939046 IP server.com.https > client.42233: Flags [S.], seq 1395986130, ack 2917118390, win 30408, options [mss 1380,sackOK,TS val 4079167253 ecr 32817878,nop,wscale 13], length 0 3. 06:06:29.033098 IP client.42233 > server.com.https: Flags [.], ack 1, win 1, options [nop,nop,TS val 32818003 ecr 4079167253], length 0 4. 06:06:29.224189 IP client.42233 > server.com.https: Flags [P.], seq 1:113, ack 1, win 1, options [nop,nop,TS val 32818194 ecr 4079167253], length 112 5. 06:06:29.255924 IP server.com.https > client.42233: Flags [.], ack 113, win 4, options [nop,nop,TS val 4079167570 ecr 32818194], length 0 6. 06:06:29.256512 IP server.com.https > client.42233: Flags [.], seq 1:1369, ack 113, win 4, options [nop,nop,TS val 4079167570 ecr 32818194], length 1368 7. 06:06:29.256520 IP server.com.https > client.42233: Flags [.], seq 1369:2737, ack 113, win 4, options [nop,nop,TS val 4079167570 ecr 32818194], length 1368 8. 06:06:29.256522 IP server.com.https > client.42233: Flags [P.], seq 2737:4097, ack 113, win 4, options [nop,nop,TS val 4079167570 ecr 32818194], length 1360 9. 06:06:29.350656 IP client.42233 > server.com.https: Flags [.], ack 1369, win 1, options [nop,nop,TS val 32818321 ecr 4079167570], length 0 10. 06:06:29.350716 IP client.42233 > server.com.https: Flags [.], ack 2737, win 1, options [nop,nop,TS val 32818321 ecr 4079167570], length 0 11. 06:06:29.350743 IP client.42233 > server.com.https: Flags [.], ack 4097, win 1, options [nop,nop,TS val 32818321 ecr 4079167570], length 0 12. 06:06:29.383781 IP server.com.https > client.42233: Flags [P.], seq 4097:4939, ack 113, win 4, options [nop,nop,TS val 4079167696 ecr 32818321], length 842 13. 06:06:29.517408 IP client.42233 > server.com.https: Flags [.], ack 4939, win 1, options [nop,nop,TS val 32818488 ecr 4079167696], length 0 14. 06:06:29.530716 IP client.42233 > server.com.https: Flags [P.], seq 113:311, ack 4939, win 1, options [nop,nop,TS val 32818501 ecr 4079167696], length 198 15. 06:06:29.564909 IP server.com.https > client.42233: Flags [P.], seq 4939:5173, ack 311, win 4, options [nop,nop,TS val 4079167879 ecr 32818501], length 234 16. 06:06:29.738429 IP client.42233 > server.com.https: Flags [.], ack 5173, win 1, options [nop,nop,TS val 32818709 ecr 4079167879], length 0 17. 06:06:33.026897 IP client.42233 > server.com.https: Flags [F.], seq 311, ack 5173, win 1, options [nop,nop,TS val 32821997 ecr 4079167879], length 0 18. 06:06:33.053611 IP server.com.https > client.42233: Flags [F.], seq 5173, ack 312, win 4, options [nop,nop,TS val 4079171372 ecr 32821997], length 0 19. 06:06:33.147658 IP client.42233 > server.com.https: Flags [.], ack 5174, win 1, options [nop,nop,TS val 32822118 ecr 4079171372], length 0 PLease note that the servers sends 4096 bytes in packets 6-8. and then waits for an ACK and only after that, it continues to send the last part of the certificate in packet 12 which unnecessary prolongs the whole handshake. This does not happen if you use smaller certificates. ''The point is that the SSL's default buffer size can be increased using BIO_set_write_buffer_size call. Could you please increase it and possibly, make it configurable?'' I double checked that the server's initial congestion window and the client's window size is not the problem. BTW: The problem is also reproducible using embedded openssl s_server." defect closed major nginx-core 1.3.x fixed SSL, " 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6 19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6 19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux" "nginx version: nginx/1.5.4 TLS SNI support enabled configure arguments: --prefix=/root/horky/nginx-root/ --without-mail_smtp_module --without-mail_imap_module --without-mail_pop3_module --without-http_auth_basic_module --without-http_autoindex_module --without-http_geo_module --without-http_map_module --without-http_fastcgi_module --without-http_scgi_module --without-http_memcached_module --without-http_empty_gif_module --with-http_ssl_module --without-http_rewrite_module --with-poll_module "