Opened 6 years ago
Last modified 6 years ago
#1607 accepted defect
mirror + limit_req = writing connections
Reported by: | Owned by: | ||
---|---|---|---|
Priority: | minor | Milestone: | |
Component: | nginx-core | Version: | 1.13.x |
Keywords: | mirror | Cc: | |
uname -a: | Linux host 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1 (2018-04-29) x86_64 GNU/Linux | ||
nginx -V: |
nginx version: nginx/1.13.12
built with OpenSSL 1.1.0f 25 May 2017 TLS SNI support enabled configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/usr/src/nginx/nginx-1.13.12=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_flv_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_secure_link_module --with-http_sub_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --add-dynamic-module=/usr/src/nginx/nginx-1.13.12/debian/modules/http-headers-more-filter --add-dynamic-module=/usr/src/nginx/nginx-1.13.12/debian/modules/http-auth-pam --add-dynamic-module=/usr/src/nginx/nginx-1.13.12/debian/modules/http-cache-purge --add-dynamic-module=/usr/src/nginx/nginx-1.13.12/debian/modules/http-dav-ext --add-dynamic-module=/usr/src/nginx/nginx-1.13.12/debian/modules/http-ndk --add-dynamic-module=/usr/src/nginx/nginx-1.13.12/debian/modules/http-echo --add-dynamic-module=/usr/src/nginx/nginx-1.13.12/debian/modules/http-fancyindex --add-dynamic-module=/usr/src/nginx/nginx-1.13.12/debian/modules/nchan --add-dynamic-module=/usr/src/nginx/nginx-1.13.12/debian/modules/http-lua --add-dynamic-module=/usr/src/nginx/nginx-1.13.12/debian/modules/rtmp --add-dynamic-module=/usr/src/nginx/nginx-1.13.12/debian/modules/http-uploadprogress --add-dynamic-module=/usr/src/nginx/nginx-1.13.12/debian/modules/http-upstream-fair --add-dynamic-module=/usr/src/nginx/nginx-1.13.12/debian/modules/http-subs-filter |
Description
Hello,
Nginx seems to have a bug with mirror+limit_req
Configuration:
All servers could be the same for testing purposes (127.0.0.1)
Frontend server
limit_req_zone $binary_remote_addr zone=one:10m rate=5r/s; location = /url1 { mirror /url2; proxy_pass http://127.0.0.1/test; } location = /url2 { internal; limit_req zone=one burst=10; proxy_pass http://127.0.0.1/test2; } location = /status { stub_status on; }
Backend server
location = /test { return 200; }
Mirror server
location = /test2 { return 200; }
Now run:
# for i in {1..1000}; do curl http://127.0.0.1/url1 >/dev/null & sleep 0.05; done
Wait for completion of all requests and see writing connections:
# curl http://127.0.0.1/status Active connections: 271 server accepts handled requests 2001 2001 2001 Reading: 0 Writing: 271 Waiting: 0 # sleep 120 # netstat -atn | grep 127.0.0.1:80 | grep -v CLOSE_WAIT | wc -l 270 # service nginx reload # pgrep -f shutting # netstat -atn | grep 127.0.0.1:80 | grep -v CLOSE_WAIT | wc -l 0 # curl http://127.0.0.1/status Active connections: 271 server accepts handled requests 2002 2002 2002 Reading: 0 Writing: 271 Waiting: 0
When /url1 doesn't have limit_req, but /url2 has, number of writing connections from stub status begins to grow. Watching netstat, I can also see CLOSE_WAIT connections growing. I did't find any impact on requests processing at least when the number of connections is quite low. Actually, after reloading nginx there seems to be no real connections (writing). But this breaks nginx monitoring. Restart of nginx only resets writing connections number.
If both /url1 and /url2 have limit_req, or /url1 only has limit_req - all is OK.
We use amd64 debian stretch, with the nginx-extras package from debian buster (rebuilt on stretch).
Change History (2)
comment:1 by , 6 years ago
comment:2 by , 6 years ago
Keywords: | mirror added |
---|---|
Status: | new → accepted |
Thanks for reporting this, I was able to reproduce this. There is a socket leak in the provided configuration because the timer as set by the limit_req in the mirror subrequest is executed in the context of the active request, and it does not wake up mirror subrequest. This needs to be addressed.
Sorry, I mean
instead of