Opened 4 years ago
Closed 4 years ago
#2130 closed defect (worksforme)
"nginx" stops every day itself
| Reported by: | Owned by: | ||
|---|---|---|---|
| Priority: | blocker | Milestone: | |
| Component: | nginx-core | Version: | 1.18.x |
| Keywords: | stopping | Cc: | |
| uname -a: | Linux test.tennis24.ee 5.4.0-64-generic #72-Ubuntu SMP Fri Jan 15 10:27:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | ||
| nginx -V: |
nginx version: nginx/1.18.0 (Ubuntu)
built with OpenSSL 1.1.1f 31 Mar 2020 TLS SNI support enabled configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-5J5hor/nginx-1.18.0=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-compat --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_flv_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_secure_link_module --with-http_sub_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --add-dynamic-module=/build/nginx-5J5hor/nginx-1.18.0/debian/modules/http-headers-more-filter --add-dynamic-module=/build/nginx-5J5hor/nginx-1.18.0/debian/modules/http-auth-pam --add-dynamic-module=/build/nginx-5J5hor/nginx-1.18.0/debian/modules/http-cache-purge --add-dynamic-module=/build/nginx-5J5hor/nginx-1.18.0/debian/modules/http-dav-ext --add-dynamic-module=/build/nginx-5J5hor/nginx-1.18.0/debian/modules/http-ndk --add-dynamic-module=/build/nginx-5J5hor/nginx-1.18.0/debian/modules/http-echo --add-dynamic-module=/build/nginx-5J5hor/nginx-1.18.0/debian/modules/http-fancyindex --add-dynamic-module=/build/nginx-5J5hor/nginx-1.18.0/debian/modules/nchan --add-dynamic-module=/build/nginx-5J5hor/nginx-1.18.0/debian/modules/http-lua --add-dynamic-module=/build/nginx-5J5hor/nginx-1.18.0/debian/modules/rtmp --add-dynamic-module=/build/nginx-5J5hor/nginx-1.18.0/debian/modules/http-uploadprogress --add-dynamic-module=/build/nginx-5J5hor/nginx-1.18.0/debian/modules/http-upstream-fair --add-dynamic-module=/build/nginx-5J5hor/nginx-1.18.0/debian/modules/http-subs-filter --add-dynamic-module=/build/nginx-5J5hor/nginx-1.18.0/debian/modules/http-geoip2 |
||
Description
Every single day, I discover that my web services are unavailable and then I check:
kalmer@test:~$ sudo systemctl status nginx
[sudo] password for kalmer:
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: failed (Result: signal) since Thu 2021-01-28 19:38:37 EET; 48s ago
Docs: man:nginx(8)
Process: 364480 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 364481 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 634407 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s reload (code=exited, status=0/SUCCESS)
Main PID: 364482 (code=killed, signal=SEGV)
Tasks: 0 (limit: 19131)
Memory: 33.9M
CGroup: /system.slice/nginx.service
jaan 28 19:38:37 test.tennis24.ee systemd[1]: nginx.service: Killing process 717850 (nginx) with signal SIGKILL.
jaan 28 19:38:37 test.tennis24.ee systemd[1]: nginx.service: Killing process 789257 (nginx) with signal SIGKILL.
jaan 28 19:38:37 test.tennis24.ee systemd[1]: nginx.service: Killing process 789258 (nginx) with signal SIGKILL.
jaan 28 19:38:37 test.tennis24.ee systemd[1]: nginx.service: Killing process 789259 (nginx) with signal SIGKILL.
jaan 28 19:38:37 test.tennis24.ee systemd[1]: nginx.service: Killing process 789260 (nginx) with signal SIGKILL.
jaan 28 19:38:37 test.tennis24.ee systemd[1]: nginx.service: Killing process 789261 (nginx) with signal SIGKILL.
jaan 28 19:38:37 test.tennis24.ee systemd[1]: nginx.service: Killing process 789262 (nginx) with signal SIGKILL.
jaan 28 19:38:37 test.tennis24.ee systemd[1]: nginx.service: Killing process 789263 (nginx) with signal SIGKILL.
jaan 28 19:38:37 test.tennis24.ee systemd[1]: nginx.service: Killing process 789264 (nginx) with signal SIGKILL.
jaan 28 19:38:37 test.tennis24.ee systemd[1]: nginx.service: Failed with result 'signal'.
kalmer@test:~$ tail /var/log/nginx/error.log
2021/01/28 09:25:24 [notice] 717826#717826: signal process started
2021/01/28 09:25:38 [notice] 717849#717849: signal process started
2021/01/28 09:50:12 [error] 717851#717851: *53151 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 194.150.65.52, server: mpd.tennis24.ee, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:1100/", host: "mpd.tennis24.ee"
2021/01/28 09:50:13 [error] 717851#717851: *53151 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 194.150.65.52, server: mpd.tennis24.ee, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:1100/favicon.ico", host: "mpd.tennis24.ee", referrer: "https://mpd.tennis24.ee/"
2021/01/28 10:36:59 [error] 717850#717850: *54277 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 217.159.151.66, server: mpd.tennis24.ee, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:1100/favicon.ico", host: "mpd.tennis24.ee", referrer: "https://mpd.tennis24.ee/"
2021/01/28 15:02:33 [error] 717850#717850: *67661 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 138.246.253.24, server: mpd.tennis24.ee, request: "HEAD / HTTP/1.1", upstream: "http://127.0.0.1:1100/", host: "mpd.tennis24.ee"
2021/01/28 15:25:53 [error] 717850#717850: *68244 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 194.150.65.10, server: mpd.tennis24.ee, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:1100/favicon.ico", host: "mpd.tennis24.ee", referrer: "https://mpd.tennis24.ee/"
2021/01/28 18:59:58 [error] 717850#717850: *73801 connect() failed (113: No route to host) while connecting to upstream, client: 104.131.114.132, server: kwamoja.tennis24.ee, request: "GET / HTTP/1.0", upstream: "http://192.168.1.65:2/", host: "kwamoja.tennis24.ee"
2021/01/28 19:38:34 [notice] 789255#789255: signal process started
2021/01/28 19:38:37 [notice] 789271#789271: signal process started
Change History (4)
comment:1 by , 4 years ago
comment:2 by , 4 years ago
On 20210213:
kalmer@test:~$ sudo systemctl status nginx
[sudo] password for kalmer:
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: failed (Result: core-dump) since Sat 2021-02-13 08:02:33 EET; 1h 2min ago
Docs: man:nginx(8)
Process: 1166 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 2074 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 2075 (code=dumped, signal=SEGV)
veebr 13 08:02:33 test.tennis24.ee systemd[1]: nginx.service: Killing process 2076 (nginx) with signal SIGKILL.
veebr 13 08:02:33 test.tennis24.ee systemd[1]: nginx.service: Killing process 71739 (nginx) with signal SIGKILL.
veebr 13 08:02:33 test.tennis24.ee systemd[1]: nginx.service: Killing process 71740 (nginx) with signal SIGKILL.
veebr 13 08:02:33 test.tennis24.ee systemd[1]: nginx.service: Killing process 71741 (nginx) with signal SIGKILL.
veebr 13 08:02:33 test.tennis24.ee systemd[1]: nginx.service: Killing process 71742 (nginx) with signal SIGKILL.
veebr 13 08:02:33 test.tennis24.ee systemd[1]: nginx.service: Killing process 71743 (nginx) with signal SIGKILL.
veebr 13 08:02:33 test.tennis24.ee systemd[1]: nginx.service: Killing process 71744 (nginx) with signal SIGKILL.
veebr 13 08:02:33 test.tennis24.ee systemd[1]: nginx.service: Killing process 71745 (nginx) with signal SIGKILL.
veebr 13 08:02:33 test.tennis24.ee systemd[1]: nginx.service: Killing process 71746 (nginx) with signal SIGKILL.
veebr 13 08:02:33 test.tennis24.ee systemd[1]: nginx.service: Failed with result 'core-dump'.
kalmer@test:~$ tail /var/log/nginx/error.log
2021/02/13 08:02:33 [notice] 71751#71751: signal process started
2021/02/13 08:02:36 [notice] 71809#71809: signal process started
2021/02/13 08:02:36 [error] 71809#71809: invalid PID number "" in "/run/nginx.pid"
2021/02/13 08:02:39 [notice] 71823#71823: signal process started
2021/02/13 08:02:42 [notice] 71838#71838: signal process started
2021/02/13 08:02:44 [alert] 71812#71812: *5 open socket #24 left in connection 6
2021/02/13 08:02:44 [alert] 71812#71812: *6 open socket #25 left in connection 7
2021/02/13 08:02:44 [alert] 71812#71812: *14 open socket #30 left in connection 9
2021/02/13 08:02:44 [alert] 71812#71812: *13 open socket #28 left in connection 10
2021/02/13 08:02:44 [alert] 71812#71812: aborting
comment:3 by , 4 years ago
As already suggested in comment:1, please provide full configuration as shown by nginx -T.
comment:4 by , 4 years ago
| Resolution: | → worksforme |
|---|---|
| Status: | new → closed |
Feedback timeout. As suggested in comment:1, most likely a configuration parsing issue in a 3rd party module.
From the logs provided it looks like nginx master process died due to SIGSEGV at about 19:38:37, likely as a result of some explicit action (note
signal process startedmessages at that time and several seconds earlier). Most likely an issue in a third party module (or nginx itself, though unlikely) during configuration parsing/reloading. Could you please provide full configuration, as shown bynginx -T?