Opened 7 years ago
Closed 7 years ago
#1153 closed defect (wontfix)
Nginx reload hang
| Reported by: | Owned by: | ||
|---|---|---|---|
| Priority: | major | Milestone: | |
| Component: | nginx-core | Version: | 1.10.x |
| Keywords: | Cc: | ||
| uname -a: | FreeBSD fbsd 9.3-RELEASE FreeBSD 9.3-RELEASE #0 r268512: Thu Jul 10 23:44:39 UTC 2014 root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 | ||
| nginx -V: |
nginx version: nginx/1.10.2
built with OpenSSL 1.0.2j 26 Sep 2016 TLS SNI support enabled configure arguments: --prefix=/usr/local/etc/nginx --with-cc-opt='-I /usr/local/include' --with-ld-opt='-L /usr/local/lib' --conf-path=/usr/local/etc/nginx/nginx.conf --sbin-path=/usr/local/sbin/nginx --pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx/error.log --user=www --group=www --modules-path=/usr/local/libexec/nginx --with-file-aio --with-ipv6 --http-client-body-temp-path=/var/tmp/nginx/client_body_temp --http-fastcgi-temp-path=/var/tmp/nginx/fastcgi_temp --http-proxy-temp-path=/var/tmp/nginx/proxy_temp --http-scgi-temp-path=/var/tmp/nginx/scgi_temp --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi_temp --http-log-path=/var/log/nginx/access.log --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gzip_static_module --with-http_gunzip_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_stub_status_module --with-http_sub_module --with-pcre --with-http_v2_module --with-stream=dynamic --with-stream_ssl_module --with-threads --with-mail=dynamic --without-mail_imap_module --without-mail_pop3_module --without-mail_smtp_module --with-mail_ssl_module --with-http_ssl_module |
||
Description
Hello,
I found a bug with fast and frequent reloading server. Tested on FreeBSD 10.3 and 9.3 with nginx 1.10.1 and 1.10.2.
How to reproduce:
# while true; do service nginx reload; done
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
(...)
The number of processes increases to 1024, like that:
PID TT STAT TIME COMMAND
10688 ?? Z 0:00.00 <defunct>
10702 ?? Z 0:00.00 <defunct>
10751 ?? Z 0:00.00 <defunct>
10800 ?? Z 0:00.00 <defunct>
10848 ?? Z 0:00.00 <defunct>
10890 ?? Z 0:00.00 <defunct>
10938 ?? Z 0:00.00 <defunct>
10986 ?? Z 0:00.00 <defunct>
11030 ?? Z 0:00.00 <defunct>
11075 ?? Z 0:00.00 <defunct>
11119 ?? Z 0:00.00 <defunct>
11166 ?? Z 0:00.00 <defunct>
11210 ?? Z 0:00.01 <defunct>
11257 ?? Z 0:00.00 <defunct>
11303 ?? Z 0:00.00 <defunct>
11350 ?? Z 0:00.00 <defunct>
11399 ?? Z 0:00.00 <defunct>
11449 ?? Z 0:00.00 <defunct>
11494 ?? Z 0:00.00 <defunct>
11539 ?? Z 0:00.00 <defunct>
11587 ?? Z 0:00.00 <defunct>
11634 ?? Z 0:00.00 <defunct>
11682 ?? Z 0:00.00 <defunct>
11726 ?? Z 0:00.00 <defunct>
11774 ?? Z 0:00.00 <defunct>
11819 ?? Z 0:00.00 <defunct>
11867 ?? Z 0:00.00 <defunct>
11916 ?? Z 0:00.00 <defunct>
11963 ?? Z 0:00.00 <defunct>
12012 ?? Z 0:00.00 <defunct>
12072 ?? Z 0:00.00 <defunct>
12118 ?? Z 0:00.00 <defunct>
12163 ?? Z 0:00.00 <defunct>
12214 ?? Z 0:00.00 <defunct>
12266 ?? Z 0:00.00 <defunct>
12318 ?? Z 0:00.00 <defunct>
12365 ?? Z 0:00.00 <defunct>
12416 ?? Z 0:00.00 <defunct>
12463 ?? Z 0:00.00 <defunct>
12515 ?? Z 0:00.00 <defunct>
12565 ?? Z 0:00.00 <defunct>
12611 ?? Z 0:00.00 <defunct>
12660 ?? Z 0:00.00 <defunct>
12704 ?? Z 0:00.00 <defunct>
12750 ?? Z 0:00.00 <defunct>
12800 ?? Z 0:00.00 <defunct>
12844 ?? Z 0:00.00 <defunct>
12894 ?? Z 0:00.00 <defunct>
12943 ?? Z 0:00.00 <defunct>
12992 ?? Z 0:00.00 <defunct>
13034 ?? Z 0:00.00 <defunct>
13083 ?? Z 0:00.00 <defunct>
13126 ?? Z 0:00.00 <defunct>
13170 ?? Z 0:00.00 <defunct>
13220 ?? S 0:00.00 nginx: worker process (nginx)
13261 ?? S 0:00.00 nginx: worker process (nginx)
When it exceed 1024 in nginx error log:
2016/12/11 13:35:07 [alert] 17134#100094: no more than 1024 processes can be spawned
2016/12/11 13:35:07 [alert] 17134#100094: no more than 1024 processes can be spawned
2016/12/11 13:35:07 [alert] 17134#100094: no more than 1024 processes can be spawned
And now nginx hangs - when I stop reloading - processes for user www are normal but nginx not reponse untill restart.
The test described exhaust available resources and sooner or later will result in problems due to resource shortage. In the particular case when nginx hits 1024 processes limit, most likely result after stopping reloading is a master process without any workers running. This happens because master wasn't able to spawn any worker processes during the last reload. In such situation new worker processes can be started by sending an additional reload signal.