id,summary,reporter,owner,description,type,status,priority,milestone,component,version,resolution,keywords,cc,uname,nginx_version 2367,nginx >= 1.17.5 returns incomplete responses for certain proxied requests during reloads,Jacob Henner,,"nginx >= 1.17.5 is returning incomplete responses for a subset of proxied requests. My observations suggest this issue occurs when: * nginx is handling a request proxied to an upstream * The upstream's response is sufficiently large (exact size undetermined, the responses in my case were > 500Mi) * The upstream's response triggers proxy buffering * proxy_buffering is enabled (does not occur when buffering is disabled) * There is a delay between the time nginx sends the request to the upstream, and the time at which the upstream returns data to nginx (e.g. the upstream service blocks on some computation before returning any data) * During the delay between nginx's proxied request and the upstream's response, an nginx reload is triggered Symptoms of the issue include: * Incomplete response bodies - for example, truncated JSON * Chunked responses where the last chunk sent does not match the chunk size specified (can be observed as a warning when `curl` runs in verbose mode). * Socket leak entries in the nginx debug logs, for both the socket connecting to the client, and the socket connecting to the upstream * Properly closed TCP connections - so no connection reset errors are propagated to the client I first observed this issue in 1.20.2. I was able to reproduce it from 1.17.5 onward (spot checked), including the current mainline 1.23.0 and stable 1.22.0 versions. All reproductions occurred in the official docker-library nginx container images. I suspect that the regression introduced in 1.17.5 that's described in https://mailman.nginx.org/pipermail/nginx-devel/2019-December/012829.html has a role here. It's my understanding that the regression was supposed to be fixed in 1.17.7, but perhaps that fix did not address this particular case. I speculate that a timer is missing, due to some issue with readv and event processing, which is causing the graceful shutdown process to prematurely terminate a worker process that's still returning buffered data to a client. Additional details: * TLS (SSL) is not in use * aio is not enabled * http2 is not enabled I have attached nginx's configuration. The socket leak log entries appear as: ``` 10867:2022/07/08 19:52:49 [notice] 8799#0: exiting 10868:2022/07/08 19:52:49 [debug] 8799#0: flush files 10869:2022/07/08 19:52:49 [alert] 8799#0: *251 open socket #4 left in connection 2 10870:2022/07/08 19:52:49 [alert] 8799#0: *252 open socket #9 left in connection 4 10871:2022/07/08 19:52:49 [alert] 8799#0: aborting ```",defect,closed,minor,,nginx-core,1.23.x,fixed,,Jacob Henner,"Linux fe6031582f6b 3.10.0-1160.15.2.el7.x86_64 #1 SMP Thu Jan 21 16:15:07 EST 2021 x86_64 GNU/Linux Linux redacted 5.10.106-102.504.amzn2.x86_64 #1 SMP Tue Mar 29 23:15:13 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux ","nginx version: nginx/1.17.5 built by gcc 8.3.0 (Debian 8.3.0-6) built with OpenSSL 1.1.1c 28 May 2019 (running with OpenSSL 1.1.1d 10 Sep 2019) TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-1.17.5/debian/debuild-base/nginx-1.17.5=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie' nginx version: nginx/1.23.0 built by gcc 10.2.1 20210110 (Debian 10.2.1-6) built with OpenSSL 1.1.1n 15 Mar 2022 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -ffile-prefix-map=/data/builder/debuild/nginx-1.23.0/debian/debuild-base/nginx-1.23.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'"