NGiNX drops the body of a HTTP proxy response on Linux if proxy doesn't read all content sent to it.
|Reported by:||Owned by:|
|uname -a:||Linux bbainfedora 4.6.4-301.fc24.x86_64 #1 SMP Tue Jul 12 11:50:00 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux|
nginx version: nginx/1.11.3
built by gcc 6.1.1 20160621 (Red Hat 6.1.1-3) (GCC)
built with OpenSSL 1.0.2h-fips 3 May 2016
TLS SNI support enabled
configure arguments: --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-http_geoip_module --with-pcre-jit --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' --prefix=/home/bbain/nginx/1.11.3 --with-debug
We have NGiNX sitting in front of gnuicorn on Linux. We have a REST resource that completely ignores the body of the request and returns a response and closes the connection. NGiNX returns the body of this resource if the size of proxy request sent (i.e. the bytes written to the socket) to gunicorn is <= 8k. If the request is >8k we end up with a HTTP response without the body.
I am pretty sure I have tracked this down. Linux will report a ECONNRESET error you try and read data on a socket that has been closed but still has outgoing data pending http://stackoverflow.com/questions/2974021/what-does-econnreset-mean-in-the-context-of-an-af-local-socket. So basically:
- Client makes connection to NGiNX.
- NGiNX send >8k worth of data to Gunicorn.
- Gunicorn reads in 8k to parse the headers. This leaves data on the socket.
- Gunicorn sends response and closes the socket leaving some data still unread.
- NGiNX gets all the response from Gunicorn.
- NGiNX gets an ECONNRESET on the next recv call.
- NGiNX assumes an error even though it has a complete response from Gunicorn and decides to only send part of the response.
The request works if in step 2 <= 8k is sent to gunicorn. In this case gunicorn will actually read all the data on the socket in its attempt to parse the headers.
I tested on my Linux VM with both 1.6.2 and 1.11.3 (which I compiled with --with-debug). This is also occurring on our production servers where we have 1.6.2 deployed.
Our setup has a unix socket between NGiNX and Gunicorn (though it also happens when using TCP sockets too). I have attached a simple test case to reproduce the problem.