Opened 9 days ago

Last modified 9 days ago

#2411 new defect

reload correlates with "open socket ... left in connection", response absent or truncated, and no access.log entry

Reported by: anderson.james.1955@… Owned by:
Priority: minor Milestone:
Component: nginx-core Version: 1.18.x
Keywords: Cc:
uname -a: Linux nl13.dydra.com 5.11.0-38-generic #42~20.04.1-Ubuntu SMP Tue Sep 28 20:41:07 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.18.0 (Ubuntu)
built with OpenSSL 1.1.1f 31 Mar 2020
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-7KvRN5/nginx-1.18.0=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-compat --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_flv_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_secure_link_module --with-http_sub_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --add-dynamic-module=/build/nginx-7KvRN5/nginx-1.18.0/debian/modules/http-headers-more-filter --add-dynamic-module=/build/nginx-7KvRN5/nginx-1.18.0/debian/modules/http-auth-pam --add-dynamic-module=/build/nginx-7KvRN5/nginx-1.18.0/debian/modules/http-cache-purge --add-dynamic-module=/build/nginx-7KvRN5/nginx-1.18.0/debian/modules/http-dav-ext --add-dynamic-module=/build/nginx-7KvRN5/nginx-1.18.0/debian/modules/http-ndk --add-dynamic-module=/build/nginx-7KvRN5/nginx-1.18.0/debian/modules/http-echo --add-dynamic-module=/build/nginx-7KvRN5/nginx-1.18.0/debian/modules/http-fancyindex --add-dynamic-module=/build/nginx-7KvRN5/nginx-1.18.0/debian/modules/nchan --add-dynamic-module=/build/nginx-7KvRN5/nginx-1.18.0/debian/modules/http-lua --add-dynamic-module=/build/nginx-7KvRN5/nginx-1.18.0/debian/modules/rtmp --add-dynamic-module=/build/nginx-7KvRN5/nginx-1.18.0/debian/modules/http-uploadprogress --add-dynamic-module=/build/nginx-7KvRN5/nginx-1.18.0/debian/modules/http-upstream-fair --add-dynamic-module=/build/nginx-7KvRN5/nginx-1.18.0/debian/modules/http-subs-filter --add-dynamic-module=/build/nginx-7KvRN5/nginx-1.18.0/debian/modules/http-geoip2

Description

we observe that curl requests to a host on which nginx serves as a reverse proxy occasionally terminate with incomplete content as nginx closes the connection.
occasionally the client has received a partial response.
the upstream server signals an i/o error as it attempts to write to a closed socket.

the nginx error log the contains entries of the sort

2022/11/17 00:26:03 [alert] 1236125#1236125: *5807696 open socket #105 left in connection 7

the occurrences follow configuration reloads.
it does not happen on every nginx reload, but each such failure is correlated with a reload.

this appears to correspond to the observations #2367, but with the addition of occurrences when no output has yet been performed.
in this installation, however, while http2 is also not enabled, ssl is.

although #2367 was closed with a corrective patch, i find no mention of the issue in release logs since that time. has it been incorporated into a release?

Change History (1)

comment:1 by Maxim Dounin, 9 days ago

The fix in question is available in nginx 1.23.1, quoting CHANGES:

    *) Bugfix: a proxied response might be truncated during reconfiguration
       on Linux; the bug had appeared in 1.17.5.

The particular bug only manifests itself when a significant number of bytes was already received by nginx from the upstream server. But due to SSL buffering being involved (up to 16k with default ssl_buffer_size), it probably can be observed as "no output has yet been performed" from the client point of view.

Please test if nginx 1.23.1 or later fixes things for you.

Note: See TracTickets for help on using tickets.