id summary reporter owner description type status priority milestone component version resolution keywords cc uname nginx_version 1026 Abrupt linger disconnect on SSL offload connections through nginx 1.10.1 on FreeBSD based proxy iamasmith.home@… "Relevant settings To ensure that chunked encoding is passed directly to the upstream server rather than being buffered until the request is complete. proxy_http_version 1.1; proxy_request_buffering off; Description I have a client that one of my developers has been working on that is getting dropped connections during large post operations if the upstream application raises an error at all. The application may produce a 409 conflict error at the beginning of a large file upload POST message from this client (OSX written using CFNetwork framework) and this causes the nginx server to enter the linger state to consume the remainder of the POST message. Unfortunately if proxying the client from an SSL connection nginx gets a 0 bytes SSL_read and returns an SSL_get_error of 5 (SSL_ERROR_SYSCALL error). This causes nginx to believe that the client has closed the connection (which it hasn't) so it cleans up and drops the connection. The ability to ensure that we aren't pooling up files being uploaded is important for this application so we need to pass the request like this without buffering it. Additional points If using plain HTTP nginx drains the lingering request properly after sending the error response to the client and the client does not get connected. The client then responds with the received error response rather than the disconnect message that one gets out of the CFNetwork framework if the request doesn't complete. It has been verified that nginx has dropped the connection by using the client software in the working HTTP mode and routing the request through STUNNEL to the nginx server which also shows that the nginx server is dropping the connection and not the client and results in the same failure. This is not observable at all until the post size reaches a good size, presumably filling any buffer/queue at the server prior to the switch to linger handling. Using a weak cipher (RC4-MD5) as the only setting I was able to analyse the traffic and confirm that following the proxy sending back the 409 to the client, the client continued to try to send chunks of data but was disconnected by the nginx proxy. I also tried adding an ERR_peek_last_error() to ngx_ssl_handle_recv if it detected SSL_ERROR_SYSCALL but this matched what was coming back as errno (0) and didn't provide any additional indication of the cause of the failure. On some occasions the linger works as expected however in most cases the connection is terminated prior to the client finishing sending. All testing has been done locally and I have experiemented with extending the linger values as a matter of course even though I can see that this isn't the problem. Test Configuration user www www; worker_processes 5; events { worker_connections 4096; debug_connection 192.168.178.1; } http { include /usr/local/etc/nginx/mime.types; default_type application/octet-stream; proxy_buffering off; large_client_header_buffers 4 12k; proxy_http_version 1.1; server { listen 192.168.178.124:443 ssl; ssl_certificate /mytest/server.crt; ssl_certificate_key /mytest/server.key; access_log /mytest/custom.log; error_log /mytest/error.log warn; underscores_in_headers on; location /api { client_max_body_size 0; keepalive_timeout 0; proxy_request_buffering off; proxy_set_header BasePath ""/api""; proxy_set_header HOST $http_host; proxy_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 1800; proxy_send_timeout 1800; proxy_pass http://127.0.0.1:8081; } } } Debug events from the disconnect. 2016/07/15 13:44:41 [debug] 13841#100298: *9 SSL_read: 4096 2016/07/15 13:44:41 [debug] 13841#100298: *9 lingering read: 4096 2016/07/15 13:44:41 [debug] 13841#100298: *9 SSL_read: 4096 2016/07/15 13:44:41 [debug] 13841#100298: *9 lingering read: 4096 2016/07/15 13:44:41 [debug] 13841#100298: *9 SSL_read: 4096 2016/07/15 13:44:41 [debug] 13841#100298: *9 lingering read: 4096 2016/07/15 13:44:41 [debug] 13841#100298: *9 SSL_read: 0 2016/07/15 13:44:41 [debug] 13841#100298: *9 SSL_get_error: 5 2016/07/15 13:44:41 [debug] 13841#100298: *9 peer shutdown SSL cleanly 2016/07/15 13:44:41 [debug] 13841#100298: *9 lingering read: 0 2016/07/15 13:44:41 [debug] 13841#100298: *9 http request count:1 blk:0 2016/07/15 13:44:41 [debug] 13841#100298: *9 http close request 2016/07/15 13:44:41 [debug] 13841#100298: *9 http log handler 2016/07/15 13:44:41 [debug] 13841#100298: *9 free: 00000008026DF000 2016/07/15 13:44:41 [debug] 13841#100298: *9 free: 000000080250D000 2016/07/15 13:44:41 [debug] 13841#100298: *9 free: 0000000802482000, unused: 1 2016/07/15 13:44:41 [debug] 13841#100298: *9 free: 000000080250C000, unused: 0 2016/07/15 13:44:41 [debug] 13841#100298: *9 free: 00000008026E0000, unused: 2949 2016/07/15 13:44:41 [debug] 13841#100298: *9 close http connection: 17 2016/07/15 13:44:41 [debug] 13841#100298: *9 SSL_shutdown: 1 2016/07/15 13:44:41 [debug] 13841#100298: *9 event timer del: 17: 1468590286811 2016/07/15 13:44:41 [debug] 13841#100298: *9 reusable connection: 0 2016/07/15 13:44:41 [debug] 13841#100298: *9 free: 0000000802487000 2016/07/15 13:44:41 [debug] 13841#100298: *9 free: 0000000802406800 2016/07/15 13:44:41 [debug] 13841#100298: *9 free: 00000008024E0800, unused: 24 2016/07/15 13:44:41 [debug] 13841#100298: *9 free: 00000008024E0E00, unused: 400 " defect closed major other 1.10.x wontfix FreeBSD testh1 10.3-RELEASE FreeBSD 10.3-RELEASE #0 r297264: Fri Mar 25 02:10:02 UTC 2016 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 "nginx version: nginx/1.10.1 built with OpenSSL 1.0.2h 3 May 2016 TLS SNI support enabled configure arguments: --prefix=/usr/local/etc/nginx --with-cc-opt='-I /usr/local/include' --with-ld-opt='-L /usr/local/lib' --conf-path=/usr/local/etc/nginx/nginx.conf --sbin-path=/usr/local/sbin/nginx --pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx-error.log --user=www --group=www --modules-path=/usr/local/libexec/nginx --with-debug --with-file-aio --with-ipv6 --http-client-body-temp-path=/var/tmp/nginx/client_body_temp --http-fastcgi-temp-path=/var/tmp/nginx/fastcgi_temp --http-proxy-temp-path=/var/tmp/nginx/proxy_temp --http-scgi-temp-path=/var/tmp/nginx/scgi_temp --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi_temp --http-log-path=/var/log/nginx-access.log --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gzip_static_module --with-http_gunzip_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_stub_status_module --with-http_sub_module --with-pcre --with-http_v2_module --with-stream=dynamic --with-stream_ssl_module --with-threads --with-mail=dynamic --without-mail_imap_module --without-mail_pop3_module --without-mail_smtp_module --with-mail_ssl_module --with-http_ssl_module"