Opened 22 months ago

Closed 22 months ago

#2363 closed defect (invalid)

(nginx) crashed and dumped core

Reported by: svarlitskiy@… Owned by:
Priority: critical Milestone:
Component: nginx-core Version: 1.23.x
Keywords: coredump, segfault, crashed Cc:
uname -a: Linux arch 5.18.6-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 22 Jun 2022 18:10:56 +0000 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.23.0
built by gcc 12.1.0 (GCC)
built with OpenSSL 1.1.1o 3 May 2022
TLS SNI support enabled
configure arguments: --prefix=/opt/nginx --with-debug --with-file-aio --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_image_filter_module --with-http_geoip_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_auth_request_module --with-http_secure_link_module --with-mail --with-mail_ssl_module --with-stream --with-stream_ssl_module --with-stream_realip_module --with-stream_geoip_module --with-compat --with-threads --with-pcre=/opt/nginx/pkgs/pcre --with-pcre-jit --with-zlib=/opt/nginx/pkgs/zlib --with-openssl=/opt/nginx/pkgs/openssl --add-module=/opt/nginx/mods/njs/nginx --add-module=/opt/nginx/mods/ngx_devel_kit --add-module=/opt/nginx/mods/ngx_brotli --add-module=/opt/nginx/mods/ngx_http_auth_pam_module --add-module=/opt/nginx/mods/ngx_dynamic_limit_req_module --add-module=/opt/nginx/mods/ngx_cookie_limit_req_module --add-module=/opt/nginx/mods/form-input-nginx-module --add-module=/opt/nginx/mods/nginx-http-auth-digest --add-module=/opt/nginx/mods/headers-more-nginx-module

Description

I am trying to run nginx and view localhost.
But I get a segfault and core dump.

I updated my os today and maybe one of the packages now causes this issue.
I will try to recompile and run.

## cat /etc/os-release

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
LOGO=archlinux-logo

## cat /usr/lib/systemd/system/nginx.service

[Unit]
Description=The NGINX HTTP and reverse proxy server
After=syslog.target network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking
PIDFile=/opt/nginx/logs/nginx.pid
ExecStartPre=/opt/nginx/sbin/nginx -t
ExecStart=/opt/nginx/sbin/nginx -g 'error_log stderr;'
ExecReload=/opt/nginx/sbin/nginx -s reload
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

## cat /opt/nginx/conf/nginx.conf

worker_processes                        auto;
worker_cpu_affinity                     auto;
worker_rlimit_nofile                    262144;

# Load modules
include                                 /opt/nginx/modules/*;

events {
    multi_accept                        on;
    worker_connections                  262144;
}

http {
    charset                             utf-8;
    sendfile                            on;
    tcp_nopush                          on;
    tcp_nodelay                         on;
    server_tokens                       off;
    log_not_found                       off;

    types_hash_max_size                 4096;
    types_hash_bucket_size              128;
    client_max_body_size                512m;
    client_body_buffer_size             24k;
    keepalive_timeout                   90s;

    proxy_headers_hash_max_size         4096;
    proxy_headers_hash_bucket_size      128;
    proxy_max_temp_file_size            512m;
    proxy_buffer_size                   24k;
    proxy_busy_buffers_size             48k;
    proxy_buffers                       64 8k;

    # MIME
    include                             mime.types;
    default_type                        application/octet-stream;

    # Logging
    access_log                          /opt/nginx/logs/access.log;
    error_log                           /opt/nginx/logs/error.log warn;

    # SSL
    ssl_ciphers                         ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_dhparam                         /opt/nginx/dhparam.pem;
    ssl_protocols                       TLSv1.3 TLSv1.2 TLSv1.1;
    ssl_session_cache                   shared:SSL:64m;
    ssl_session_tickets                 off;
    ssl_session_timeout                 12h;
    ssl_stapling                        on;
    ssl_stapling_verify                 on;

    resolver_timeout                    2s;
    resolver                            1.1.1.1 1.0.0.1 8.8.8.8 8.8.4.4 9.9.9.9 149.112.112.112 valid=30s;

    # Default index types
    index                               index.html index.htm index.php index.vue index.ahtml index.bhtml index.chtml index.dhtml index.ehtml index.fhtml index.ghtml index.hhtml index.ihtml index.jhtml index.khtml index.lhtml index.mhtml index.nhtml index.ohtml index.phtml index.qhtml index.rhtml index.shtml index.thtml index.uhtml index.vhtml index.whtml index.xhtml index.yhtml index.zhtml;

    # Connection header for WebSocket reverse proxy
    map $http_upgrade $connection_upgrade {
        default                         upgrade;
        ""                              close;
    }

    # Add For Variable
    map $remote_addr $proxy_forwarded_elem {
        ~^[0-9.]+$                     "for=$remote_addr";
        ~^[0-9A-Fa-f:.]+$              "for=\"[$remote_addr]\"";
        default                         "for=unknown";
    }

    # Add Forwarded header if syntactically valid or replace it
    map $http_forwarded $proxy_add_forwarded {
        "~^(,[ \\t]*)*([!#$%&'*+.^_`|~0-9A-Za-z-]+=([!#$%&'*+.^_`|~0-9A-Za-z-]+|\"([\\t \\x21\\x23-\\x5B\\x5D-\\x7E\\x80-\\xFF]|\\\\[\\t \\x21-\\x7E\\x80-\\xFF])*\"))?(;([!#$%&'*+.^_`|~0-9A-Za-z-]+=([!#$%&'*+.^_`|~0-9A-Za-z-]+|\"([\\t \\x21\\x23-\\x5B\\x5D-\\x7E\\x80-\\xFF]|\\\\[\\t \\x21-\\x7E\\x80-\\xFF])*\"))?)*([ \\t]*,([ \\t]*([!#$%&'*+.^_`|~0-9A-Za-z-]+=([!#$%&'*+.^_`|~0-9A-Za-z-]+|\"([\\t \\x21\\x23-\\x5B\\x5D-\\x7E\\x80-\\xFF]|\\\\[\\t \\x21-\\x7E\\x80-\\xFF])*\"))?(;([!#$%&'*+.^_`|~0-9A-Za-z-]+=([!#$%&'*+.^_`|~0-9A-Za-z-]+|\"([\\t \\x21\\x23-\\x5B\\x5D-\\x7E\\x80-\\xFF]|\\\\[\\t \\x21-\\x7E\\x80-\\xFF])*\"))?)*)?)*$" "$http_forwarded, $proxy_forwarded_elem";
        default                         "$proxy_forwarded_elem";
    }

    # Enable PHP Requests
    include                             fastcgi.conf;
    upstream php_backend {
        least_conn;
        zone                        php_backend 64K;
        server                      127.0.0.1:9000 max_fails=1 fail_timeout=2s;
        keepalive                   32;
    }

    # Enable Security Configs
    add_header      X-XSS-Protection             "1; mode=block" always;
    add_header      X-Content-Type-Options       "nosniff" always;
    add_header      Referrer-Policy              "no-referrer-when-downgrade" always;
    add_header      Content-Security-Policy      "upgrade-insecure-requests; default-src 'none'; base-uri 'none'; object-src 'none'; trusted-types 'none'; child-src 'self' blob: ; connect-src 'self' https: http: blob: mediastream: filesystem: ; font-src 'self' blob: https: ; form-action 'self' blob: https: ; frame-ancestors 'self' blob: https: ; frame-src 'self' blob: https: ; img-src 'self' blob: https: ; manifest-src 'self' blob: ; media-src 'self' blob: https: mediastream: ; navigate-to 'self' https: http: blob: ; prefetch-src 'self' blob: https: ; script-src 'unsafe-hashes' 'unsafe-inline' 'self' blob: https: ; style-src 'unsafe-inline' 'self' blob: https: ; worker-src 'self' blob: mediastream: ; sandbox allow-downloads allow-forms allow-popups-to-escape-sandbox allow-same-origin allow-scripts;" always;

    # Localhost
    server {
        listen                          80;
        listen                          [::]:80;

        server_name                             localhost 127.0.0.1;
        root                                    /var/www/localhost/live;

        location /api/ {
            proxy_pass              http://127.0.0.1:3000;

            proxy_set_header        Host                 $host;
            proxy_set_header        X-Real-IP            $remote_addr;
            proxy_set_header        X-Forwarded-For      $proxy_add_x_forwarded_for;
            proxy_set_header        Upgrade              $http_upgrade;
            proxy_set_header        Connection           "upgrade";

            proxy_http_version      1.1;
        }

        location ~ \.php$ {
            fastcgi_pass                php_backend;
            fastcgi_keep_conn           on;
            fastcgi_index               index.php;
        }

        location / {
            try_files               $uri $uri/ /index.html;
        }

        brotli                          on;
        brotli_comp_level               8;
        brotli_types                    text/plain text/css text/xml application/json application/javascript application/rss+xml application/atom+xml image/svg+xml;

        gzip                            on;
        gzip_vary                       on;
        gzip_proxied                    any;
        gzip_comp_level                 8;
        gzip_types                      text/plain text/css text/xml application/json application/javascript application/rss+xml application/atom+xml image/svg+xml;

        error_page   404   @ERR_404;
        location @ERR_404 {
            auth_basic off;
            allow all;
            try_files       /404.html /40X.html /4XX.html =404;
        }
    }

}


## sudo systemctl status nginx

● nginx.service - The NGINX HTTP and reverse proxy server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled)
     Active: active (running) since Wed 2022-06-29 11:54:08 PDT; 48min ago
    Process: 23887 ExecStartPre=/opt/nginx/sbin/nginx -t (code=exited, status=0/SUCCESS)
    Process: 23888 ExecStart=/opt/nginx/sbin/nginx -g error_log stderr; (code=exited, status=0/SUCCESS)
   Main PID: 23889 (nginx)
      Tasks: 9 (limit: 38438)
     Memory: 940.3M
        CPU: 1.174s
     CGroup: /system.slice/nginx.service
             ├─23889 "nginx: master process /opt/nginx/sbin/nginx -g error_log stderr;"
             ├─23891 "nginx: worker process"
             ├─23892 "nginx: worker process"
             ├─23893 "nginx: worker process"
             ├─23894 "nginx: worker process"
             ├─23895 "nginx: worker process"
             ├─23896 "nginx: worker process"
             ├─23897 "nginx: worker process"
             └─23957 "nginx: worker process"

Jun 29 11:54:08 arch systemd[1]: Starting The NGINX HTTP and reverse proxy server...
Jun 29 11:54:08 arch nginx[23887]: nginx: the configuration file /opt/nginx/conf/nginx.conf syntax is ok
Jun 29 11:54:08 arch nginx[23887]: nginx: configuration file /opt/nginx/conf/nginx.conf test is successful
Jun 29 11:54:08 arch systemd[1]: Started The NGINX HTTP and reverse proxy server.
Jun 29 11:54:38 arch nginx[23889]: 2022/06/29 11:54:38 [alert] 23889#23889: worker process 23890 exited on signal 11 (core dumped)

## sudo journalctl --no-pager -xe -n 100

Jun 29 11:54:08 arch systemd[1]: Started The NGINX HTTP and reverse proxy server.
░░ Subject: A start job for unit nginx.service has finished successfully
░░ Defined-By: systemd
░░ 
░░ A start job for unit nginx.service has finished successfully.
░░ 
░░ The job identifier is 4589.
Jun 29 11:54:08 arch audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=nginx comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 29 11:54:08 arch kernel: audit: type=1130 audit(1656528848.275:1731): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=nginx comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 29 11:54:08 arch sudo[23884]: pam_unix(sudo:session): session closed for user root
Jun 29 11:54:08 arch audit[23884]: USER_END pid=23884 uid=1000 auid=1000 ses=4 subj==unconfined msg='op=PAM:session_close grantors=pam_systemd_home,pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
Jun 29 11:54:08 arch audit[23884]: CRED_DISP pid=23884 uid=1000 auid=1000 ses=4 subj==unconfined msg='op=PAM:setcred grantors=pam_faillock,pam_permit,pam_env,pam_faillock acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
Jun 29 11:54:12 arch dbus-daemon[531]: [system] Activating via systemd: service name='org.freedesktop.home1' unit='dbus-org.freedesktop.home1.service' requested by ':1.333' (uid=0 pid=23898 comm="sudo journalctl --no-pager -xe -n 200")
Jun 29 11:54:12 arch dbus-daemon[531]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.home1.service': Unit dbus-org.freedesktop.home1.service not found.
Jun 29 11:54:12 arch sudo[23898]: pam_systemd_home(sudo:account): systemd-homed is not available: Unit dbus-org.freedesktop.home1.service not found.
Jun 29 11:54:12 arch audit[23898]: USER_ACCT pid=23898 uid=1000 auid=1000 ses=4 subj==unconfined msg='op=PAM:accounting grantors=pam_unix,pam_permit,pam_time acct="master" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
Jun 29 11:54:12 arch audit[23898]: CRED_REFR pid=23898 uid=1000 auid=1000 ses=4 subj==unconfined msg='op=PAM:setcred grantors=pam_faillock,pam_permit,pam_env,pam_faillock acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
Jun 29 11:54:12 arch sudo[23898]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Jun 29 11:54:12 arch audit[23898]: USER_START pid=23898 uid=1000 auid=1000 ses=4 subj==unconfined msg='op=PAM:session_open grantors=pam_systemd_home,pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
Jun 29 11:54:12 arch kernel: kauditd_printk_skb: 2 callbacks suppressed
Jun 29 11:54:12 arch kernel: audit: type=1101 audit(1656528852.165:1734): pid=23898 uid=1000 auid=1000 ses=4 subj==unconfined msg='op=PAM:accounting grantors=pam_unix,pam_permit,pam_time acct="master" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
Jun 29 11:54:12 arch kernel: audit: type=1110 audit(1656528852.165:1735): pid=23898 uid=1000 auid=1000 ses=4 subj==unconfined msg='op=PAM:setcred grantors=pam_faillock,pam_permit,pam_env,pam_faillock acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
Jun 29 11:54:12 arch kernel: audit: type=1105 audit(1656528852.165:1736): pid=23898 uid=1000 auid=1000 ses=4 subj==unconfined msg='op=PAM:session_open grantors=pam_systemd_home,pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
Jun 29 11:54:12 arch sudo[23898]: pam_unix(sudo:session): session closed for user root
Jun 29 11:54:12 arch audit[23898]: USER_END pid=23898 uid=1000 auid=1000 ses=4 subj==unconfined msg='op=PAM:session_close grantors=pam_systemd_home,pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
Jun 29 11:54:12 arch audit[23898]: CRED_DISP pid=23898 uid=1000 auid=1000 ses=4 subj==unconfined msg='op=PAM:setcred grantors=pam_faillock,pam_permit,pam_env,pam_faillock acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
Jun 29 11:54:12 arch kernel: audit: type=1106 audit(1656528852.188:1737): pid=23898 uid=1000 auid=1000 ses=4 subj==unconfined msg='op=PAM:session_close grantors=pam_systemd_home,pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
Jun 29 11:54:12 arch kernel: audit: type=1104 audit(1656528852.188:1738): pid=23898 uid=1000 auid=1000 ses=4 subj==unconfined msg='op=PAM:setcred grantors=pam_faillock,pam_permit,pam_env,pam_faillock acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
Jun 29 11:54:37 arch audit[23890]: ANOM_ABEND auid=4294967295 uid=962 gid=962 ses=4294967295 subj==unconfined pid=23890 comm="nginx" exe="/opt/nginx/sbin/nginx" sig=11 res=1
Jun 29 11:54:37 arch kernel: nginx[23890]: segfault at 0 ip 00007f951177bdbb sp 00007ffcc4ad5290 error 4 in libc.so.6[7f9511761000+178000]
Jun 29 11:54:37 arch kernel: Code: 89 d7 48 89 74 24 10 85 c9 0f 85 b0 02 00 00 83 ff 01 0f 84 97 00 00 00 83 ff 24 0f 87 8e 00 00 00 48 8b 5c 24 08 49 8b 48 68 <48> 0f be 13 48 89 d0 f6 44 51 01 20 74 1a 0f 1f 80 00 00 00 00 48
Jun 29 11:54:37 arch kernel: audit: type=1701 audit(1656528877.868:1739): auid=4294967295 uid=962 gid=962 ses=4294967295 subj==unconfined pid=23890 comm="nginx" exe="/opt/nginx/sbin/nginx" sig=11 res=1
Jun 29 11:54:37 arch audit: BPF prog-id=203 op=LOAD
Jun 29 11:54:37 arch audit: BPF prog-id=204 op=LOAD
Jun 29 11:54:37 arch audit: BPF prog-id=205 op=LOAD
Jun 29 11:54:37 arch systemd[1]: Started Process Core Dump (PID 23948/UID 0).
░░ Subject: A start job for unit systemd-coredump@32-23948-0.service has finished successfully
░░ Defined-By: systemd
░░ 
░░ A start job for unit systemd-coredump@32-23948-0.service has finished successfully.
░░ 
░░ The job identifier is 4688.
Jun 29 11:54:37 arch audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=systemd-coredump@32-23948-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 29 11:54:37 arch kernel: audit: type=1334 audit(1656528877.892:1740): prog-id=203 op=LOAD
Jun 29 11:54:37 arch kernel: audit: type=1334 audit(1656528877.892:1741): prog-id=204 op=LOAD
Jun 29 11:54:37 arch kernel: audit: type=1334 audit(1656528877.892:1742): prog-id=205 op=LOAD
Jun 29 11:54:37 arch kernel: audit: type=1130 audit(1656528877.892:1743): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=systemd-coredump@32-23948-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 29 11:54:38 arch systemd-coredump[23949]: [man:core(5)] Process 23890 (nginx) of user 962 dumped core.
                                              
                                              Module linux-vdso.so.1 with build-id 0a9dd56f2d1d7d248eeef56d29e78f6fddab09d4
                                              Module libnss_systemd.so.2 with build-id 78182715685be902c90ae8edb29ddd056486a8ae
                                              Module libpcre.so.1 with build-id 845483dd0acba86de9f0313102bebbaf3ce52767
                                              Module libmvec.so.1 with build-id 2c6aec81c79e91d66940b68be0819fdf780a976c
                                              Module libXdmcp.so.6 with build-id d864159ab0008415667db8d5f251696d75c90df2
                                              Module libXau.so.6 with build-id 60db1eac70f819bea9d4c366603c1583067510b4
                                              Module libglib-2.0.so.0 with build-id f1d15261ce1317b9003a1f0957d5a528d063f630
                                              Module libgraphite2.so.3 with build-id ce58945ebb55b86d3a4e717b6eae29efc4720d8e
                                              Module libyuv.so with build-id 60a987609f24d8125baf7f8f2a3b288eb465d87e
                                              Module libSvtAv1Enc.so.1 with build-id d95f643a00090915d6f17c27a84a031c45e819f3
                                              Module librav1e.so.0 with build-id 5f8fc2c271f0d96480ab1c71447e038a66aa1c57
                                              Module libdav1d.so.6 with build-id a93560a23a7f1def734a3cf54f09eadd7c3c46e9
                                              Module libgcc_s.so.1 with build-id 0e3de903950e35ae59a5de8c00b1817a4a71ca01
                                              Module libstdc++.so.6 with build-id b3eaebcb5369487562e52dd8d016c9cd77f80c54
                                              Module libx265.so.199 with build-id 4fd2d7155725df753c6b32274c442349fc05e632
                                              Module libde265.so.0 with build-id 459cf009e847aef8f1589ad39b1573351b6a3b22
                                              Module libaom.so.3 with build-id 2b5b58e482957529f8b14aecb5b2acc21362d04d
                                              Module libpthread.so.0 with build-id 95ae4f30a6f12ccbff645d30f8e1a3ee23ec7d36
                                              Module liblzma.so.5 with build-id 28b40c7af8098a66af6ee093b6986b91cad7694d
                                              Module libzstd.so.1 with build-id ab54c2881f53ab314e134f3e08c76d504376dd5d
                                              Module libxcb.so.1 with build-id 13d677412a71468381b11092915d231f664d18d3
                                              Module libbrotlidec.so.1 with build-id 66c54e9301f7e102ecc1d88547e5f0e8a056fe22
                                              Module libharfbuzz.so.0 with build-id 4b8338b259318d7786bb1f4a702954f2e74fb3dd
                                              Module libbz2.so.1.0 with build-id 919597c477c9b2cb9cdbb7745ed6494ac0e6da60
                                              Module libexpat.so.1 with build-id 113bb5a3e9ad856801bfcfc029102c9bdc13d67e
                                              Module libcap-ng.so.0 with build-id 707f9d3134a43306625e3dab8662899ea368ac91
                                              Module libavif.so.14 with build-id 7c7c60390f804291a41ed3e75cd75121eda0ed35
                                              Module libheif.so.1 with build-id 64bccee060b5f2b512bc5ab529a4030b4f58ea8b
                                              Module libwebp.so.7 with build-id 0add7a572705244ed810b7e722cce7da0dcfb62c
                                              Module libtiff.so.5 with build-id 31895d2bd133f34f0cdc2d4ac855ed838ec927b6
                                              Module libX11.so.6 with build-id 310f8f1b8d7457d326407e0eb794353c62769b73
                                              Module libXpm.so.4 with build-id f6892c81626c982958f96c1f757d52210382ea9a
                                              Module libjpeg.so.8 with build-id 8e6d3f3e8f438912b561c43b6e7f66e6e5e097d0
                                              Module libfreetype.so.6 with build-id f89dd5502e75aca28fb5c3ccd0dbd26fe822bfef
                                              Module libfontconfig.so.1 with build-id 36be6951b8c1e42a7dd05684a37400fc8ef9147c
                                              Module libz.so.1 with build-id fefe3219a96d682ec98fcfb78866b8594298b5a2
                                              Module libpng16.so.16 with build-id 2dc0bce07f199bf983c07a05fb95a6f4af83a9b3
                                              Module libdl.so.2 with build-id 94198b268228074fa9f405bbedbbae94112593ed
                                              Module libaudit.so.1 with build-id 53c797475ffeed05918d78f49f85dbf4127fb174
                                              Module libbrotlicommon.so.1 with build-id acfd597a977c8087bb6184383daae2e828a9ce42
                                              Module ld-linux-x86-64.so.2 with build-id 0effd0e43efa4468d3c31871c93af0b7f3005673
                                              Module libc.so.6 with build-id 60df1df31f02a7b23da83e8ef923359885b81492
                                              Module libGeoIP.so.1 with build-id 5d444ed5635d931e0a595d4cace64d223bbc3aca
                                              Module libgd.so.3 with build-id 6f138001d21f218bd558016c023d76de94db3969
                                              Module libhiredis.so.1.0.0 with build-id 2421bd3130ca487cdf1a55d1569bb69e73b74c79
                                              Module libpam.so.0 with build-id bb11b2685fe89555938ffd330ea44d82b0f8701c
                                              Module libbrotlienc.so.1 with build-id 74adbc62e4fbb5da9d37b5aa458471f4130862ff
                                              Module libm.so.6 with build-id 1b7296ef9fd806e47060788389293c824b09ad72
                                              Module libcrypt.so.2 with build-id a0a45f81771945f0559d04e93726d245159930da
                                              Module nginx with build-id c39d56290da5f913dcb4ef37c5e05d2710265299
                                              Stack trace of thread 23890:
                                              #0  0x00007f951177bdbb n/a (libc.so.6 + 0x42dbb)
                                              #1  0x0000559175d2e2d9 atoi (nginx + 0x1c42d9)
                                              #2  0x0000559175c796b3 ngx_http_core_generic_phase (nginx + 0x10f6b3)
                                              #3  0x0000559175c74b87 ngx_http_core_run_phases (nginx + 0x10ab87)
                                              #4  0x0000559175c74c2a ngx_http_handler (nginx + 0x10ac2a)
                                              #5  0x0000559175c81068 ngx_http_process_request (nginx + 0x117068)
                                              #6  0x0000559175c816be ngx_http_process_request_headers (nginx + 0x1176be)
                                              #7  0x0000559175c81aa5 ngx_http_process_request_line (nginx + 0x117aa5)
                                              #8  0x0000559175c81c78 ngx_http_wait_request_handler (nginx + 0x117c78)
                                              #9  0x0000559175c64155 ngx_epoll_process_events (nginx + 0xfa155)
                                              #10 0x0000559175c5849c ngx_process_events_and_timers (nginx + 0xee49c)
                                              #11 0x0000559175c61efb ngx_worker_process_cycle (nginx + 0xf7efb)
                                              #12 0x0000559175c6038c ngx_spawn_process (nginx + 0xf638c)
                                              #13 0x0000559175c60ff4 ngx_start_worker_processes (nginx + 0xf6ff4)
                                              #14 0x0000559175c62807 ngx_master_process_cycle (nginx + 0xf8807)
                                              #15 0x0000559175c369a6 main (nginx + 0xcc9a6)
                                              #16 0x00007f9511762290 n/a (libc.so.6 + 0x29290)
                                              #17 0x00007f951176234a __libc_start_main (libc.so.6 + 0x2934a)
                                              #18 0x0000559175c35065 _start (nginx + 0xcb065)
                                              ELF object binary architecture: AMD x86-64
░░ Subject: Process 23890 (nginx) dumped core
░░ Defined-By: systemd
░░ Documentation: man:core(5)
░░ 
░░ Process 23890 (nginx) crashed and dumped core.
░░ 
░░ This usually indicates a programming error in the crashing program and
░░ should be reported to its vendor as a bug.
Jun 29 11:54:38 arch systemd[1]: systemd-coredump@32-23948-0.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ 
░░ The unit systemd-coredump@32-23948-0.service has successfully entered the 'dead' state.
Jun 29 11:54:38 arch kernel: audit: type=1131 audit(1656528878.168:1744): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=systemd-coredump@32-23948-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 29 11:54:38 arch audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=systemd-coredump@32-23948-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 29 11:54:38 arch nginx[23889]: 2022/06/29 11:54:38 [alert] 23889#23889: worker process 23890 exited on signal 11 (core dumped)
Jun 29 11:54:38 arch audit: BPF prog-id=0 op=UNLOAD
Jun 29 11:54:38 arch audit: BPF prog-id=0 op=UNLOAD
Jun 29 11:54:38 arch audit: BPF prog-id=0 op=UNLOAD
Jun 29 11:54:38 arch kernel: audit: type=1334 audit(1656528878.298:1745): prog-id=0 op=UNLOAD
Jun 29 11:54:38 arch kernel: audit: type=1334 audit(1656528878.298:1746): prog-id=0 op=UNLOAD
Jun 29 11:54:38 arch kernel: audit: type=1334 audit(1656528878.298:1747): prog-id=0 op=UNLOAD
Jun 29 11:54:42 arch dbus-daemon[531]: [system] Activating via systemd: service name='org.freedesktop.home1' unit='dbus-org.freedesktop.home1.service' requested by ':1.334' (uid=0 pid=23959 comm="sudo journalctl --no-pager -xe -n 200")
Jun 29 11:54:42 arch dbus-daemon[531]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.home1.service': Unit dbus-org.freedesktop.home1.service not found.
Jun 29 11:54:42 arch sudo[23959]: pam_systemd_home(sudo:account): systemd-homed is not available: Unit dbus-org.freedesktop.home1.service not found.
Jun 29 11:54:42 arch audit[23959]: USER_ACCT pid=23959 uid=1000 auid=1000 ses=4 subj==unconfined msg='op=PAM:accounting grantors=pam_unix,pam_permit,pam_time acct="master" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
Jun 29 11:54:42 arch audit[23959]: CRED_REFR pid=23959 uid=1000 auid=1000 ses=4 subj==unconfined msg='op=PAM:setcred grantors=pam_faillock,pam_permit,pam_env,pam_faillock acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
Jun 29 11:54:42 arch audit[23959]: USER_START pid=23959 uid=1000 auid=1000 ses=4 subj==unconfined msg='op=PAM:session_open grantors=pam_systemd_home,pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'

Change History (3)

comment:1 by Maxim Dounin, 22 months ago

Are you able to reproduce this without 3rd party modules? Note that there are major API changes in nginx 1.23.0, and 3rd party modules which aren't properly updated can cause coredumps.

Also, quick look suggests that at least "ngx_dynamic_limit_req_module" and "ngx_cookie_limit_req_module" are likely to segfault regardless of the API changes. You may want to start with removing both modules to see if it helps.

comment:2 by svarlitskiy@…, 22 months ago

I think there was something that changed in either the lib packages or linux kernel that caused it to stop working. I was able to get things working by:

  1. Recompiling nginx with all the 3rd party packages
  2. Restarting the entire computer
  • Note: A restart is needed, will not work without it. 
## cat /var/log/pacman.log

[2022-06-29T08:08:08-0700] [ALPM] upgraded libblockdev (2.26-3 -> 2.27-1)
[2022-06-29T08:08:08-0700] [ALPM] upgraded libidn (1.40-1 -> 1.41-1)
[2022-06-29T08:08:08-0700] [ALPM] upgraded libmm-glib (1.18.8-1 -> 1.18.10-1)
[2022-06-29T08:08:08-0700] [ALPM] upgraded libnghttp2 (1.47.0-1 -> 1.48.0-1)
[2022-06-29T08:08:08-0700] [ALPM] upgraded liburing (2.1-1 -> 2.2-1)
[2022-06-29T08:08:08-0700] [ALPM] upgraded linux (5.18.6.arch1-1 -> 5.18.7.arch1-1)
[2022-06-29T08:08:08-0700] [ALPM] upgraded linux-headers (5.18.6.arch1-1 -> 5.18.7.arch1-1)

comment:3 by Maxim Dounin, 22 months ago

Resolution: → invalid
Status: new → closed

Well, it was definitely an issue with 3rd party modules. As already suggested, most likely with either ngx_dynamic_limit_req_module or ngx_cookie_limit_req_module: both call atoi() at the preaccess phase, which matches the stack trace provided. If it now works for you, this might indicate that either relevant bug was fixed, or something changed in your configuration so it is no longer triggered.

Just in case, I would recommend to recompile nginx without both mentioned modules even if they seem to work. Both seems to try to connect Redis even if not configured, and both use blocking calls to Redis, so will block the entire nginx worker process if there are network issues.

Either way, good it now works for you. Closing this.

Note: See TracTickets for help on using tickets.