Opened 8 years ago

Closed 8 years ago

#1153 closed defect (wontfix)

Nginx reload hang

Reported by: avkarenow@… Owned by:
Priority: major Milestone:
Component: nginx-core Version: 1.10.x
Keywords: Cc:
uname -a: FreeBSD fbsd 9.3-RELEASE FreeBSD 9.3-RELEASE #0 r268512: Thu Jul 10 23:44:39 UTC 2014 root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
nginx -V: nginx version: nginx/1.10.2
built with OpenSSL 1.0.2j 26 Sep 2016
TLS SNI support enabled
configure arguments: --prefix=/usr/local/etc/nginx --with-cc-opt='-I /usr/local/include' --with-ld-opt='-L /usr/local/lib' --conf-path=/usr/local/etc/nginx/nginx.conf --sbin-path=/usr/local/sbin/nginx --pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx/error.log --user=www --group=www --modules-path=/usr/local/libexec/nginx --with-file-aio --with-ipv6 --http-client-body-temp-path=/var/tmp/nginx/client_body_temp --http-fastcgi-temp-path=/var/tmp/nginx/fastcgi_temp --http-proxy-temp-path=/var/tmp/nginx/proxy_temp --http-scgi-temp-path=/var/tmp/nginx/scgi_temp --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi_temp --http-log-path=/var/log/nginx/access.log --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gzip_static_module --with-http_gunzip_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_stub_status_module --with-http_sub_module --with-pcre --with-http_v2_module --with-stream=dynamic --with-stream_ssl_module --with-threads --with-mail=dynamic --without-mail_imap_module --without-mail_pop3_module --without-mail_smtp_module --with-mail_ssl_module --with-http_ssl_module

Description

Hello,

I found a bug with fast and frequent reloading server. Tested on FreeBSD 10.3 and 9.3 with nginx 1.10.1 and 1.10.2.

How to reproduce:
# while true; do service nginx reload; done
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
(...)

The number of processes increases to 1024, like that:

PID TT STAT TIME COMMAND

10688 ?? Z 0:00.00 <defunct>
10702 ?? Z 0:00.00 <defunct>
10751 ?? Z 0:00.00 <defunct>
10800 ?? Z 0:00.00 <defunct>
10848 ?? Z 0:00.00 <defunct>
10890 ?? Z 0:00.00 <defunct>
10938 ?? Z 0:00.00 <defunct>
10986 ?? Z 0:00.00 <defunct>
11030 ?? Z 0:00.00 <defunct>
11075 ?? Z 0:00.00 <defunct>
11119 ?? Z 0:00.00 <defunct>
11166 ?? Z 0:00.00 <defunct>
11210 ?? Z 0:00.01 <defunct>
11257 ?? Z 0:00.00 <defunct>
11303 ?? Z 0:00.00 <defunct>
11350 ?? Z 0:00.00 <defunct>
11399 ?? Z 0:00.00 <defunct>
11449 ?? Z 0:00.00 <defunct>
11494 ?? Z 0:00.00 <defunct>
11539 ?? Z 0:00.00 <defunct>
11587 ?? Z 0:00.00 <defunct>
11634 ?? Z 0:00.00 <defunct>
11682 ?? Z 0:00.00 <defunct>
11726 ?? Z 0:00.00 <defunct>
11774 ?? Z 0:00.00 <defunct>
11819 ?? Z 0:00.00 <defunct>
11867 ?? Z 0:00.00 <defunct>
11916 ?? Z 0:00.00 <defunct>
11963 ?? Z 0:00.00 <defunct>
12012 ?? Z 0:00.00 <defunct>
12072 ?? Z 0:00.00 <defunct>
12118 ?? Z 0:00.00 <defunct>
12163 ?? Z 0:00.00 <defunct>
12214 ?? Z 0:00.00 <defunct>
12266 ?? Z 0:00.00 <defunct>
12318 ?? Z 0:00.00 <defunct>
12365 ?? Z 0:00.00 <defunct>
12416 ?? Z 0:00.00 <defunct>
12463 ?? Z 0:00.00 <defunct>
12515 ?? Z 0:00.00 <defunct>
12565 ?? Z 0:00.00 <defunct>
12611 ?? Z 0:00.00 <defunct>
12660 ?? Z 0:00.00 <defunct>
12704 ?? Z 0:00.00 <defunct>
12750 ?? Z 0:00.00 <defunct>
12800 ?? Z 0:00.00 <defunct>
12844 ?? Z 0:00.00 <defunct>
12894 ?? Z 0:00.00 <defunct>
12943 ?? Z 0:00.00 <defunct>
12992 ?? Z 0:00.00 <defunct>
13034 ?? Z 0:00.00 <defunct>
13083 ?? Z 0:00.00 <defunct>
13126 ?? Z 0:00.00 <defunct>
13170 ?? Z 0:00.00 <defunct>
13220 ?? S 0:00.00 nginx: worker process (nginx)
13261 ?? S 0:00.00 nginx: worker process (nginx)

When it exceed 1024 in nginx error log:
2016/12/11 13:35:07 [alert] 17134#100094: no more than 1024 processes can be spawned
2016/12/11 13:35:07 [alert] 17134#100094: no more than 1024 processes can be spawned
2016/12/11 13:35:07 [alert] 17134#100094: no more than 1024 processes can be spawned

And now nginx hangs - when I stop reloading - processes for user www are normal but nginx not reponse untill restart.

Change History (1)

comment:1 by Maxim Dounin, 8 years ago

Resolution: wontfix
Status: newclosed

The test described exhaust available resources and sooner or later will result in problems due to resource shortage. In the particular case when nginx hits 1024 processes limit, most likely result after stopping reloading is a master process without any workers running. This happens because master wasn't able to spawn any worker processes during the last reload. In such situation new worker processes can be started by sending an additional reload signal.

Note: See TracTickets for help on using tickets.