Opened 5 years ago

Closed 5 years ago

#1822 closed defect (invalid)

segfault when running nginx in reverse TLS proxy configuration on Raspberry Pi Zero W

Reported by: antonsviridenko@… Owned by:
Priority: major Milestone:
Component: other Version: 1.17.x
Keywords: Cc:
uname -a: Linux meye-c3a0d767 4.19.49 #1 Sun Jul 14 19:43:01 +04 2019 armv6l GNU/Linux
nginx -V: nginx version: nginx/1.17.2
built by gcc 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)
built with OpenSSL 1.1.1a 20 Nov 2018
TLS SNI support enabled
configure arguments: --crossbuild=Linux::arm --with-cc=/home/odsk/motioneye_raspberry/motioneyeos/output/raspberrypi/host/bin/arm-linux-gnueabihf-gcc --with-cpp=/home/odsk/motioneye_raspberry/motioneyeos/output/raspberrypi/host/bin/arm-linux-gnueabihf-gcc --with-ld-opt= --prefix=/usr --conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/sbin/nginx --pid-path=/var/run/nginx.pid --lock-path=/var/run/lock/nginx.lock --user=www-data --group=www-data --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/tmp/nginx/client-body --http-proxy-temp-path=/var/tmp/nginx/proxy --http-fastcgi-temp-path=/var/tmp/nginx/fastcgi --http-scgi-temp-path=/var/tmp/nginx/scgi --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi --with-pcre --without-select_module --without-poll_module --without-http-cache --with-http_ssl_module --with-cc-opt='-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g2 -D_FORTIFY_SOURCE=1'

Description

nginx version 1.17.2 is built by "motioneye OS" (based on Buildroot),
crashes on Raspberry Pi Zero W.
Does not crash when compiled with --debug config option, so I am unable to attach verbose debug log.

stack trace:

Program received signal SIGSEGV, Segmentation fault.
0x0082a284 in ?? ()
(gdb) bt full
#0  0x0082a284 in ?? ()
No symbol table info available.
#1  0x0006898c in ngx_http_upstream_process_header (r=0x829210, u=0x829ac0) at src/http/ngx_http_upstream.c:2387
        n = <optimized out>
        rc = <optimized out>
        c = 0x80f578
#2  0x00066fb4 in ngx_http_upstream_handler (ev=<optimized out>) at src/http/ngx_http_upstream.c:1286
        c = 0x80f4f8
        r = <optimized out>
        u = <optimized out>
#3  0x00044ab4 in ngx_epoll_process_events (cycle=<optimized out>, timer=<optimized out>, flags=<optimized out>) at src/event/modules/ngx_epoll_module.c:902
        events = 1
        revents = 281268
        instance = <optimized out>
        i = 0
        level = <optimized out>
        err = <optimized out>
        rev = <optimized out>
        wev = <optimized out>
        queue = <optimized out>
        c = 0x80f578
#4  0x0003bdb4 in ngx_process_events_and_timers (cycle=cycle@entry=0x8093f0) at src/event/ngx_event.c:242
        flags = <optimized out>
        timer = <optimized out>
        delta = 60860280
#5  0x00043250 in ngx_worker_process_cycle (cycle=0x8093f0, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:750
        worker = <optimized out>
#6  0x0004183c in ngx_spawn_process (cycle=cycle@entry=0x8093f0, proc=0x0, data=0xbea9e728, name=0x97530 "worker process", respawn=respawn@entry=0) at src/os/unix/ngx_process.c:199
        on = 1
        pid = 0
        s = 0
#7  0x0004400c in ngx_reap_children (cycle=0x8093f0) at src/os/unix/ngx_process_cycle.c:622
        i = 0
        live = <optimized out>
        n = <optimized out>
        ch = {command = 2, pid = 24896, slot = 0, fd = -1}
        ccf = <optimized out>
        i = <optimized out>
        n = <optimized out>
        live = <optimized out>
        ch = <optimized out>
        ccf = <optimized out>
#8  ngx_master_process_cycle (cycle=cycle@entry=0x8093f0) at src/os/unix/ngx_process_cycle.c:175
        title = <optimized out>
        p = <optimized out>
        size = <optimized out>
        i = <optimized out>
        n = <optimized out>
---Type <return> to continue, or q <return> to quit---
        sigio = 0
        set = {__val = {0 <repeats 32 times>}}
        itv = {it_interval = {tv_sec = 603436, tv_usec = 0}, it_value = {tv_sec = 0, tv_usec = 0}}
        live = <optimized out>
        delay = 838148
        ls = <optimized out>
        ccf = 0x0
#9  0x0001d8cc in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:382
        b = <optimized out>
        log = 0x0
        i = <optimized out>
        cycle = 0x8093f0
        init_cycle = {conf_ctx = 0x0, pool = 0x8072c0, log = 0xba394 <ngx_log>, new_log = {log_level = 0, file = 0x0, connection = 0, disk_full_time = 0, handler = 0x0, data = 0x0, writer = 0x0, wdata = 0x0, 
            action = 0x0, next = 0x0}, log_use_stderr = 0, files = 0x0, free_connections = 0x0, free_connection_n = 0, modules = 0x0, modules_n = 0, modules_used = 0, reusable_connections_queue = {prev = 0x0, 
            next = 0x0}, reusable_connections_n = 0, listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, paths = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, config_dump = {
            elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, config_dump_rbtree = {root = 0x0, sentinel = 0x0, insert = 0x0}, config_dump_sentinel = {key = 0, left = 0x0, right = 0x0, parent = 0x0, 
            color = 0 '\000', data = 0 '\000'}, open_files = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts = 0x0, 
              nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, connection_n = 0, files_n = 0, connections = 0x0, read_events = 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {len = 26, 
            data = 0xbea9ec55 "ss"}, conf_param = {len = 0, data = 0x0}, conf_prefix = {len = 16, data = 0xbea9ec55 "ss"}, prefix = {len = 5, data = 0x93fd0 "/usr/"}, lock_file = {len = 0, data = 0x0}, 
          hostname = {len = 0, data = 0x0}}
        cd = <optimized out>
        ccf = <optimized out>
(gdb) quit

config

#user  nobody;
worker_processes  2;

error_log  /var/log/nginx/error.log info;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  24;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        off;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    # HTTPS server
    #
    server {
        listen       443 ssl;
        server_name  motioneye.picapica.im;

        ssl_certificate      /data/etc/nginx/cert.pem;
        ssl_certificate_key  /data/etc/nginx/cert.key;

        ssl_session_cache    shared:SSL:1m;
        ssl_session_timeout  5m;

        ssl_ciphers  HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers  on;

        location /cams/ {
        proxy_pass http://localhost:80/;
        proxy_read_timeout 120s;
        access_log off;
    }
    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    }

error log:

2019/07/28 10:09:34 [notice] 24881#0: using the "epoll" event method
2019/07/28 10:09:34 [notice] 24881#0: nginx/1.17.2
2019/07/28 10:09:34 [notice] 24881#0: built by gcc 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611) 
2019/07/28 10:09:34 [notice] 24881#0: OS: Linux 4.19.49
2019/07/28 10:09:34 [notice] 24881#0: getrlimit(RLIMIT_NOFILE): 1024:4096
2019/07/28 10:09:34 [notice] 24882#0: start worker processes
2019/07/28 10:09:34 [notice] 24882#0: start worker process 24883
2019/07/28 10:09:34 [notice] 24882#0: start worker process 24884
2019/07/28 10:09:43 [notice] 24882#0: signal 17 (SIGCHLD) received from 24884
2019/07/28 10:09:43 [alert] 24882#0: worker process 24884 exited on signal 11
2019/07/28 10:09:43 [notice] 24882#0: start worker process 24889
2019/07/28 10:09:43 [notice] 24882#0: signal 29 (SIGIO) received
2019/07/28 10:09:44 [notice] 24882#0: signal 17 (SIGCHLD) received from 24883
2019/07/28 10:09:44 [alert] 24882#0: worker process 24883 exited on signal 11
2019/07/28 10:09:44 [notice] 24882#0: start worker process 24890
2019/07/28 10:09:44 [notice] 24882#0: signal 29 (SIGIO) received
2019/07/28 10:09:44 [notice] 24882#0: signal 17 (SIGCHLD) received from 24889
2019/07/28 10:09:44 [alert] 24882#0: worker process 24889 exited on signal 11
2019/07/28 10:09:44 [notice] 24882#0: start worker process 24891
2019/07/28 10:09:44 [notice] 24882#0: signal 29 (SIGIO) received
2019/07/28 10:09:44 [notice] 24882#0: signal 17 (SIGCHLD) received from 24890
2019/07/28 10:09:44 [alert] 24882#0: worker process 24890 exited on signal 11
2019/07/28 10:09:44 [notice] 24882#0: start worker process 24892
2019/07/28 10:09:44 [notice] 24882#0: signal 29 (SIGIO) received
2019/07/28 10:09:45 [notice] 24882#0: signal 17 (SIGCHLD) received from 24891
2019/07/28 10:09:45 [alert] 24882#0: worker process 24891 exited on signal 11
2019/07/28 10:09:45 [notice] 24882#0: start worker process 24893
2019/07/28 10:09:45 [notice] 24882#0: signal 29 (SIGIO) received
2019/07/28 10:09:45 [notice] 24882#0: signal 17 (SIGCHLD) received from 24892
2019/07/28 10:09:45 [alert] 24882#0: worker process 24892 exited on signal 11
2019/07/28 10:09:45 [notice] 24882#0: start worker process 24894
2019/07/28 10:09:45 [notice] 24882#0: signal 29 (SIGIO) received
2019/07/28 10:09:45 [notice] 24882#0: signal 17 (SIGCHLD) received from 24893
2019/07/28 10:09:45 [alert] 24882#0: worker process 24893 exited on signal 11
2019/07/28 10:09:45 [notice] 24882#0: start worker process 24895
2019/07/28 10:09:45 [notice] 24882#0: signal 29 (SIGIO) received
2019/07/28 10:09:46 [notice] 24882#0: signal 17 (SIGCHLD) received from 24894
2019/07/28 10:09:46 [alert] 24882#0: worker process 24894 exited on signal 11
2019/07/28 10:09:46 [notice] 24882#0: start worker process 24896
2019/07/28 10:09:46 [notice] 24882#0: signal 29 (SIGIO) received
2019/07/28 10:09:46 [notice] 24882#0: signal 17 (SIGCHLD) received from 24895
2019/07/28 10:09:46 [alert] 24882#0: worker process 24895 exited on signal 11
2019/07/28 10:09:46 [notice] 24882#0: start worker process 24897
2019/07/28 10:09:46 [notice] 24882#0: signal 29 (SIGIO) received
2019/07/28 10:09:46 [notice] 24882#0: signal 17 (SIGCHLD) received from 24896
2019/07/28 10:09:46 [alert] 24882#0: worker process 24896 exited on signal 11
2019/07/28 10:09:46 [notice] 24882#0: start worker process 24898
2019/07/28 10:09:46 [notice] 24882#0: signal 29 (SIGIO) received
2019/07/28 10:10:25 [info] 24898#0: epoll_wait() failed (4: Interrupted system call)
2019/07/28 10:10:32 [notice] 24882#0: signal 17 (SIGCHLD) received from 24897
2019/07/28 10:10:32 [alert] 24882#0: worker process 24897 exited on signal 11
2019/07/28 10:10:32 [notice] 24882#0: start worker process 24948
2019/07/28 10:10:32 [notice] 24882#0: signal 29 (SIGIO) received
2019/07/28 10:12:54 [notice] 24882#0: signal 17 (SIGCHLD) received from 24898
2019/07/28 10:12:54 [alert] 24882#0: worker process 24898 exited on signal 11
2019/07/28 10:12:54 [notice] 24882#0: start worker process 25053
2019/07/28 10:12:54 [notice] 24882#0: signal 29 (SIGIO) received
2019/07/28 10:12:54 [notice] 24882#0: signal 17 (SIGCHLD) received from 24948
2019/07/28 10:12:54 [alert] 24882#0: worker process 24948 exited on signal 11

Change History (2)

comment:1 by Maxim Dounin, 5 years ago

We've seen number of problems with both Buildroot-compiled nginx - due to Buildroot's cross-compilation patches for nginx, see #899. Please recompile nginx without any patches, natively, and report if the problem persists.

Note well that we've also observed miscompilation by gcc 4.9 on Raspberry Pi, see #912. As such, I would recommend testing compilation without -O2.

comment:2 by Maxim Dounin, 5 years ago

Resolution: → invalid
Status: new → closed

Feedback timeout. Likely a crossbuild issue.

Note: See TracTickets for help on using tickets.