#1135 closed defect (invalid)
Connections timing out after upgrading to 1.10.2
| Reported by: | Owned by: | ||
|---|---|---|---|
| Priority: | critical | Milestone: | |
| Component: | nginx-core | Version: | 1.10.x |
| Keywords: | memstore | Cc: | |
| uname -a: | Linux host1.swd.local 4.4.0-45-generic #66~14.04.1-Ubuntu SMP Wed Oct 19 15:05:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | ||
| nginx -V: |
nginx version: nginx/1.10.2
built with OpenSSL 1.0.2j 26 Sep 2016 TLS SNI support enabled configure arguments: --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_spdy_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_flv_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_secure_link_module --with-http_sub_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-stream=dynamic --with-stream_ssl_module --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/headers-more-nginx-module --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/nginx-auth-pam --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/nginx-cache-purge --add-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/nginx-dav-ext-module --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/nginx-development-kit --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/nginx-echo --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/ngx-fancyindex --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/nchan --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/nginx-lua --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/nginx-upload-progress --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/nginx-upstream-fair --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/ngx_http_substitutions_filter_module |
||
Description
After upgrading from 1.10.1 without ALPN support to 1.10.2 with ALPN support on Ubuntu Trusty (this ppa: https://launchpad.net/~ondrej/+archive/ubuntu/nginx), we've been getting into situations where Nginx completely stops serving connections without any warning.
This has been happening on multiple production hosts a less than a week after the upgrade. The nginx error log on the affected hosts gets these odd messages:
2016/11/19 09:04:21 [alert] 26819#26819: worker process 28247 exited on signal 6 (core dumped) 2016/11/19 09:04:21 [alert] 26819#26819: shared memory zone "memstore" was locked by 28247 ter process /usr/sbin/nginx: /build/nginx-silDYy/nginx-1.10.2/debian/modules/nchan/src/store/memory/memstore.c:670: nchan_store_init_worker: Assertion `procslot_found == 1' failed. 2016/11/19 09:04:21 [alert] 26819#26819: worker process 28251 exited on signal 6 (core dumped) 2016/11/19 09:04:21 [alert] 26819#26819: shared memory zone "memstore" was locked by 28251 ter process /usr/sbin/nginx: /build/nginx-silDYy/nginx-1.10.2/debian/modules/nchan/src/store/memory/memstore.c:670: nchan_store_init_worker: Assertion `procslot_found == 1' failed. 2016/11/19 09:04:21 [alert] 26819#26819: worker process 28252 exited on signal 6 (core dumped) 2016/11/19 09:04:21 [alert] 26819#26819: shared memory zone "memstore" was locked by 28252 ter process /usr/sbin/nginx: /build/nginx-silDYy/nginx-1.10.2/debian/modules/nchan/src/store/memory/memstore.c:670: nchan_store_init_worker: Assertion `procslot_found == 1' failed. 2016/11/19 09:04:21 [alert] 26819#26819: worker process 28249 exited on signal 6 (core dumped) 2016/11/19 09:04:21 [alert] 26819#26819: shared memory zone "memstore" was locked by 28249
After restarting Nginx, everything works okay again.
Thank you for your assistance!
Change History (5)
comment:1 by , 7 years ago
comment:2 by , 7 years ago
| Resolution: | → invalid |
|---|---|
| Status: | new → closed |
According to the error log message, this issue is in the 3rd-party module. Please, report any problems related to 3rd-party modules to their authors.
follow-up: 4 comment:3 by , 7 years ago
Getting rid of mod-nchan hasn't helped. Still getting these issues.
comment:4 by , 7 years ago
Replying to anttiviljami@…:
Getting rid of mod-nchan hasn't helped. Still getting these issues.
Please, try to reproduce the issue without other 3rd-party modules and provide the debug log.
comment:5 by , 7 years ago
I've been hit by this as well. Disabling the nchan module worked for me. Just remove the conf file under modules-enabled dir did the trick. At least now I can't reproduce the issue
Nchan has a somewhat similar report related to nchan crashing when nginx tries to reload: https://github.com/slact/nchan/issues/129
You could try upgrading to latest version 1.10.2-2+deb.sury.org~trusty+1 published by Ondrej on Nov 16th or simply try if the problem go away if you remove nchan?