Opened 8 years ago

Closed 7 years ago

#1097 closed defect (fixed)

Nginx worker process exited on signal 11

Reported by: adushein.sports.ru@… Owned by:
Priority: major Milestone:
Component: nginx-core Version: 1.10.x
Keywords: Cc:
uname -a: FreeBSD fe1.sports.ru 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11 18:38:15 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
nginx -V: nginx version: nginx/1.10.1
built with OpenSSL 1.0.2h 3 May 2016
TLS SNI support enabled
configure arguments: --prefix=/usr/local/etc/nginx --with-cc-opt='-I /usr/local/include' --with-ld-opt='-L /usr/local/lib' --conf-path=/usr/local/etc/nginx/nginx.conf --sbin-path=/usr/local/sbin/nginx --pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx/error.log --user=www --group=www --with-file-aio --http-client-body-temp-path=/var/tmp/nginx/client_body_temp --http-fastcgi-temp-path=/var/tmp/nginx/fastcgi_temp --http-proxy-temp-path=/var/tmp/nginx/proxy_temp --http-scgi-temp-path=/var/tmp/nginx/scgi_temp --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi_temp --http-log-path=/var/log/nginx/access.log --with-http_dav_module --with-http_geoip_module --with-http_gzip_static_module --with-http_realip_module --with-http_slice_module --with-http_stub_status_module --with-http_sub_module --with-pcre --with-http_v2_module --with-stream --with-stream_ssl_module --with-http_ssl_module

Description (last modified by Maxim Dounin)

I have 2 nginx frontends worked in load balancing mode.
Today, around the same time on both frontends I got error: worker process exited on signal 11
Backtrace for both core dumps are similar. Backtrace are below.

#0  0x0000000000000000 in ?? ()
No symbol table info available.
#1  0x0000000000457a37 in ngx_kqueue_process_events (cycle=0x8024b3050, timer=23, flags=1) at src/event/modules/ngx_kqueue_module.c:669
        events = 512
        n = 85
        i = 501
        instance = 0
        level = 140737488349120
        err = 0
        ev = (ngx_event_t *) 0x806bd7478
        queue = (ngx_queue_t *) 0x4ab0b1
        ts = {tv_sec = 0, tv_nsec = 23000000}
        tp = (struct timespec *) 0x7fffffffe758
#2  0x00000000004482fe in ngx_process_events_and_timers (cycle=0x8024b3050) at src/event/ngx_event.c:242
        flags = 1
        timer = 23
        delta = 1475766275958
#3  0x0000000000455aa3 in ngx_worker_process_cycle (cycle=0x8024b3050, data=0x3) at src/os/unix/ngx_process_cycle.c:753
        worker = 3
#4  0x0000000000451d30 in ngx_spawn_process (cycle=0x8024b3050, proc=0x4559d0 <ngx_worker_process_cycle>, data=0x3,
    name=0x523cd6 "worker process", respawn=-4) at src/os/unix/ngx_process.c:198
        on = 1
        pid = 0
        s = 20
#5  0x0000000000453a29 in ngx_start_worker_processes (cycle=0x8024b3050, n=16, type=-4) at src/os/unix/ngx_process_cycle.c:358
        i = 3
        ch = {command = 1, pid = 40518, slot = 19, fd = 8}
#6  0x00000000004537be in ngx_master_process_cycle (cycle=0x8024b3050) at src/os/unix/ngx_process_cycle.c:243
        title = 0x8097e7b9c "\b"
        p = (u_char *) 0x8097e7bc0 "\020"
        size = 37
        i = 1
        n = 140737488349776
        sigio = 0
        set = {__bits = {0, 0, 0, 0}}
        itv = {it_interval = {tv_sec = 0, tv_usec = 0}, it_value = {tv_sec = 6, tv_usec = 0}}
        live = 1
        delay = 0
        ls = (ngx_listening_t *) 0x0
        ccf = (ngx_core_conf_t *) 0x8024b4b58
#7  0x00000000004129f8 in main (argc=1, argv=0x7fffffffed60) at src/core/nginx.c:367
        b = (ngx_buf_t *) 0x800745c00
        log = (ngx_log_t *) 0x75b648
        i = 34386931443
        cycle = (ngx_cycle_t *) 0x802483050
        init_cycle = {conf_ctx = 0x0, pool = 0x802406800, log = 0x75b648, new_log = {log_level = 0, file = 0x0, connection = 0,
    disk_full_time = 0, handler = 0, data = 0x0, writer = 0, wdata = 0x0, action = 0x0, next = 0x0}, log_use_stderr = 0, files = 0x0,
  free_connections = 0x0, free_connection_n = 0, modules = 0x0, modules_n = 0, modules_used = 0, reusable_connections_queue = {prev = 0x0,
    next = 0x0}, listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, paths = {elts = 0x0, nelts = 0, size = 0,
    nalloc = 0, pool = 0x0}, config_dump = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, open_files = {last = 0x0, part = {
      elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0,
      next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, connection_n = 0, files_n = 0, connections = 0x0, read_events = 0x0,
  write_events = 0x0, old_cycle = 0x0, conf_file = {len = 31, data = 0x520358 "/usr/local/etc/nginx/nginx.conf"}, conf_param = {len = 0,
    data = 0x0}, conf_prefix = {len = 21, data = 0x520358 "/usr/local/etc/nginx/nginx.conf"}, prefix = {len = 21,
    data = 0x520342 "/usr/local/etc/nginx/"}, lock_file = {len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}
        cd = (ngx_conf_dump_t *) 0x80076a000
        ccf = (ngx_core_conf_t *) 0x802484d08

Change History (4)

comment:1 by Maxim Dounin, 8 years ago

Description: modified (diff)

comment:2 by Maxim Dounin, 8 years ago

Please provide output of the following gdb commands:

fr 1
p *ev
p nevents
p i
p event_list[i]
p event_list[i-1]

Please also provide full configuration used (can be easily obtained with nginx -T).

in reply to:  description comment:3 by begracyber@…, 7 years ago

Replying to adushein.sports.ru@…:

I have 2 nginx frontends worked in load balancing mode.
Today, around the same time on both frontends I got error: worker process exited on signal 11
Backtrace for both core dumps are similar. Backtrace are below.

#0  0x0000000000000000 in ?? ()
No symbol table info available.
#1  0x0000000000457a37 in ngx_kqueue_process_events (cycle=0x8024b3050, timer=23, flags=1) at src/event/modules/ngx_kqueue_module.c:669
        events = 512
        n = 85
        i = 501
        instance = 0
        level = 140737488349120
        err = 0
        ev = (ngx_event_t *) 0x806bd7478
        queue = (ngx_queue_t *) 0x4ab0b1
        ts = {tv_sec = 0, tv_nsec = 23000000}
        tp = (struct timespec *) 0x7fffffffe758
#2  0x00000000004482fe in ngx_process_events_and_timers (cycle=0x8024b3050) at src/event/ngx_event.c:242
        flags = 1
        timer = 23
        delta = 1475766275958
#3  0x0000000000455aa3 in ngx_worker_process_cycle (cycle=0x8024b3050, data=0x3) at src/os/unix/ngx_process_cycle.c:753
        worker = 3
#4  0x0000000000451d30 in ngx_spawn_process (cycle=0x8024b3050, proc=0x4559d0 <ngx_worker_process_cycle>, data=0x3,
    name=0x523cd6 "worker process", respawn=-4) at src/os/unix/ngx_process.c:198
        on = 1
        pid = 0
        s = 20
#5  0x0000000000453a29 in ngx_start_worker_processes (cycle=0x8024b3050, n=16, type=-4) at src/os/unix/ngx_process_cycle.c:358
        i = 3
        ch = {command = 1, pid = 40518, slot = 19, fd = 8}
#6  0x00000000004537be in ngx_master_process_cycle (cycle=0x8024b3050) at src/os/unix/ngx_process_cycle.c:243
        title = 0x8097e7b9c "\b"
        p = (u_char *) 0x8097e7bc0 "\020"
        size = 37
        i = 1
        n = 140737488349776
        sigio = 0
        set = {__bits = {0, 0, 0, 0}}
        itv = {it_interval = {tv_sec = 0, tv_usec = 0}, it_value = {tv_sec = 6, tv_usec = 0}}
        live = 1
        delay = 0
        ls = (ngx_listening_t *) 0x0
        ccf = (ngx_core_conf_t *) 0x8024b4b58
#7  0x00000000004129f8 in main (argc=1, argv=0x7fffffffed60) at src/core/nginx.c:367
        b = (ngx_buf_t *) 0x800745c00
        log = (ngx_log_t *) 0x75b648
        i = 34386931443
        cycle = (ngx_cycle_t *) 0x802483050
        init_cycle = {conf_ctx = 0x0, pool = 0x802406800, log = 0x75b648, new_log = {log_level = 0, file = 0x0, connection = 0,
    disk_full_time = 0, handler = 0, data = 0x0, writer = 0, wdata = 0x0, action = 0x0, next = 0x0}, log_use_stderr = 0, files = 0x0,
  free_connections = 0x0, free_connection_n = 0, modules = 0x0, modules_n = 0, modules_used = 0, reusable_connections_queue = {prev = 0x0,
    next = 0x0}, listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, paths = {elts = 0x0, nelts = 0, size = 0,
    nalloc = 0, pool = 0x0}, config_dump = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, open_files = {last = 0x0, part = {
      elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0,
      next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, connection_n = 0, files_n = 0, connections = 0x0, read_events = 0x0,
  write_events = 0x0, old_cycle = 0x0, conf_file = {len = 31, data = 0x520358 "/usr/local/etc/nginx/nginx.conf"}, conf_param = {len = 0,
    data = 0x0}, conf_prefix = {len = 21, data = 0x520358 "/usr/local/etc/nginx/nginx.conf"}, prefix = {len = 21,
    data = 0x520342 "/usr/local/etc/nginx/"}, lock_file = {len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}
        cd = (ngx_conf_dump_t *) 0x80076a000
        ccf = (ngx_core_conf_t *) 0x802484d08

comment:4 by Maxim Dounin, 7 years ago

Resolution: → fixed
Status: new → closed

Since no requested details are provided it is not possible to tell what exactly goes wrong here, though I suspect it is most likely the aio + X-Accel-Redirect problem as mentioned in 5850fed24639. Closing this as "fixed" accordingly.

Note: See TracTickets for help on using tickets.