Opened 8 years ago

Closed 8 years ago

Last modified 8 years ago

#1135 closed defect (invalid)

Connections timing out after upgrading to 1.10.2

Reported by: anttiviljami@… Owned by:
Priority: critical Milestone:
Component: nginx-core Version: 1.10.x
Keywords: memstore Cc:
uname -a: Linux host1.swd.local 4.4.0-45-generic #66~14.04.1-Ubuntu SMP Wed Oct 19 15:05:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.10.2
built with OpenSSL 1.0.2j 26 Sep 2016
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_spdy_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_flv_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_secure_link_module --with-http_sub_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-stream=dynamic --with-stream_ssl_module --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/headers-more-nginx-module --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/nginx-auth-pam --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/nginx-cache-purge --add-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/nginx-dav-ext-module --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/nginx-development-kit --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/nginx-echo --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/ngx-fancyindex --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/nchan --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/nginx-lua --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/nginx-upload-progress --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/nginx-upstream-fair --add-dynamic-module=/build/nginx-silDYy/nginx-1.10.2/debian/modules/ngx_http_substitutions_filter_module

Description

After upgrading from 1.10.1 without ALPN support to 1.10.2 with ALPN support on Ubuntu Trusty (this ppa: https://launchpad.net/~ondrej/+archive/ubuntu/nginx), we've been getting into situations where Nginx completely stops serving connections without any warning.

This has been happening on multiple production hosts a less than a week after the upgrade. The nginx error log on the affected hosts gets these odd messages:

2016/11/19 09:04:21 [alert] 26819#26819: worker process 28247 exited on signal 6 (core dumped)
2016/11/19 09:04:21 [alert] 26819#26819: shared memory zone "memstore" was locked by 28247
ter process /usr/sbin/nginx: /build/nginx-silDYy/nginx-1.10.2/debian/modules/nchan/src/store/memory/memstore.c:670: nchan_store_init_worker: Assertion `procslot_found == 1' failed.
2016/11/19 09:04:21 [alert] 26819#26819: worker process 28251 exited on signal 6 (core dumped)
2016/11/19 09:04:21 [alert] 26819#26819: shared memory zone "memstore" was locked by 28251
ter process /usr/sbin/nginx: /build/nginx-silDYy/nginx-1.10.2/debian/modules/nchan/src/store/memory/memstore.c:670: nchan_store_init_worker: Assertion `procslot_found == 1' failed.
2016/11/19 09:04:21 [alert] 26819#26819: worker process 28252 exited on signal 6 (core dumped)
2016/11/19 09:04:21 [alert] 26819#26819: shared memory zone "memstore" was locked by 28252
ter process /usr/sbin/nginx: /build/nginx-silDYy/nginx-1.10.2/debian/modules/nchan/src/store/memory/memstore.c:670: nchan_store_init_worker: Assertion `procslot_found == 1' failed.
2016/11/19 09:04:21 [alert] 26819#26819: worker process 28249 exited on signal 6 (core dumped)
2016/11/19 09:04:21 [alert] 26819#26819: shared memory zone "memstore" was locked by 28249

After restarting Nginx, everything works okay again.

Thank you for your assistance!

Change History (5)

comment:1 by otto.seravo.fi@…, 8 years ago

Nchan has a somewhat similar report related to nchan crashing when nginx tries to reload: https://github.com/slact/nchan/issues/129

You could try upgrading to latest version 1.10.2-2+deb.sury.org~trusty+1 published by Ondrej on Nov 16th or simply try if the problem go away if you remove nchan?

sudo apt remove libnginx-mod-nchan


comment:2 by Valentin V. Bartenev, 8 years ago

Resolution: invalid
Status: newclosed

According to the error log message, this issue is in the 3rd-party module. Please, report any problems related to 3rd-party modules to their authors.

comment:3 by anttiviljami@…, 8 years ago

Getting rid of mod-nchan hasn't helped. Still getting these issues.

in reply to:  3 comment:4 by Valentin V. Bartenev, 8 years ago

Replying to anttiviljami@…:

Getting rid of mod-nchan hasn't helped. Still getting these issues.

Please, try to reproduce the issue without other 3rd-party modules and provide the debug log.

comment:5 by juan.brein.breins.net@…, 8 years ago

I've been hit by this as well. Disabling the nchan module worked for me. Just remove the conf file under modules-enabled dir did the trick. At least now I can't reproduce the issue

Note: See TracTickets for help on using tickets.