nginx+ssl crashes very often on low memory environment

[Michael Blue]

Hello,

I'm using nginx version 1.6.1 stable branch and I'm experiencing few hundred crashes/day. First I thought that the problem was caused by ngx_pagespeed, so, I stopped using that. Seems like the bug appeared after I began using ssl certificates, especially in low memory/stress conditions.

openssl  version
OpenSSL 1.0.1i 6 Aug 2014

pcre-config --version
8.35

free -m
             total       used       free     shared    buffers     cached
Mem:          1011        937         73        296         11        463
-/+ buffers/cache:        462        548
Swap:         1375        392        983

Traces:

GNU gdb (Debian 7.7.1+dfsg-3) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i586-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/nginx-1.6.1-nopagespeed/sbin/nginx...done.
[New LWP 19233]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Core was generated by `nginx: worker process      '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  ngx_ssl_new_session (ssl_conn=0x8fbf3d8, sess=0x8fbb4f8) at src/event/ngx_event_openssl.c:1890
1890	    cache = shm_zone->data;
(gdb) bt full
#0  ngx_ssl_new_session (ssl_conn=0x8fbf3d8, sess=0x8fbb4f8) at src/event/ngx_event_openssl.c:1890
        len = 140
        p = 0xbf939c98 "\001"
        id = <optimized out>
        cached_sess = <optimized out>
        ssl_ctx = 0x8d4ed00
        shm_zone = 0x0
        c = 0xb051d628
        shpool = <optimized out>
        sess_id = <optimized out>
        cache = <optimized out>
        buf = "0\201\211\002\001\001\002\002\003\001\004\002\000\071\004 \211\016\070\031\201\061\253\031M\373\233Wj\303Z\217\b\341\b\n&*)\273\222\267\063\225\313B\217E\004\060\204\365ن\254`\bt\302\305%z\376\230\211\023\256k~\306\362E}g\rG\226@\250*\355\334\377\331*\336\305\362\231\313\360\022]a3\230C\372\241\006\002\004S\373\021U\242\004\002\002\002X\244\006\004\004HTTP\246\022\004\020egcdn.gazduit.ro\001\000\000\000\004", '\000' <repeats 11 times>, "\363\235\223\277\000\000\000 \000\000\000\000 $s\267X\237\223\277\065\211*\267\375\235\223\277@?\374\b@?\374\b\300?\374\b\270\271\373\b\215\237\031_"...
#1  0xb7637c7c in ?? () from /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0
No symbol table info available.
#2  0xb760f684 in ?? () from /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0
No symbol table info available.
#3  0xb763871d in SSL_do_handshake () from /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0
No symbol table info available.
#4  0x0806ef0e in ngx_ssl_handshake (c=0xb051d628) at src/event/ngx_event_openssl.c:785
        n = <optimized out>
        sslerr = <optimized out>
        err = <optimized out>
#5  0x0806f27c in ngx_ssl_handshake_handler (ev=0x8e758b8) at src/event/ngx_event_openssl.c:933
        c = 0xb051d628
#6  0x080642a2 in ngx_event_process_posted (cycle=0x8d413b8, posted=0x810694c <ngx_posted_events>) at src/event/ngx_event_posted.c:40
        ev = 0x8e758b8
#7  0x08063e83 in ngx_process_events_and_timers (cycle=0x8d413b8) at src/event/ngx_event.c:275
        flags = 3
        timer = <optimized out>
        delta = 443
#8  0x0806a558 in ngx_worker_process_cycle (cycle=0x8d413b8, data=0x1) at src/os/unix/ngx_process_cycle.c:816
        worker = <optimized out>
        i = <optimized out>
        c = <optimized out>
#9  0x08068fa0 in ngx_spawn_process (cycle=0x8d413b8, proc=0x806a493 <ngx_worker_process_cycle>, data=0x1, name=0x80c444a "worker process", respawn=1) at src/os/unix/ngx_process.c:198
        on = 1
        pid = 0
        s = 1
#10 0x0806b06c in ngx_reap_children (cycle=0x8d413b8) at src/os/unix/ngx_process_cycle.c:627
        i = <optimized out>
        live = 1
        n = <optimized out>
        ch = {command = 2, pid = 19219, slot = 1, fd = -1}
        ccf = <optimized out>
#11 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:180
        title = <optimized out>
        p = <optimized out>
        size = <optimized out>
        i = <optimized out>
        n = <optimized out>
        sigio = 0
        set = {__val = {0 <repeats 32 times>}}
        itv = {it_interval = {tv_sec = 842261476, tv_usec = 170998070}, it_value = {tv_sec = -1217107664, tv_usec = 1}}
        live = <optimized out>
        delay = 0
        ls = <optimized out>
---Type <return> to continue, or q <return> to quit---
        ccf = 0x8d41c84
#12 0x0804de3e in main (argc=1, argv=0xbf93b174) at src/core/nginx.c:407
        i = <optimized out>
        log = <optimized out>
        cycle = 0x8d413b8
        init_cycle = {conf_ctx = 0x0, pool = 0x8d40cd0, log = 0x8104660 <ngx_log>, new_log = {log_level = 0, file = 0x0, connection = 0, handler = 0x0, data = 0x0, action = 0x0, next = 0x0}, log_use_stderr = 0, files = 0x0, 
          free_connections = 0x0, free_connection_n = 0, reusable_connections_queue = {prev = 0x0, next = 0x0}, listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, paths = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, 
            pool = 0x0}, open_files = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, 
            pool = 0x0}, connection_n = 0, files_n = 0, connections = 0x0, read_events = 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {len = 50, data = 0x8d40cf8 "\002"}, conf_param = {len = 0, data = 0x0}, conf_prefix = {
            len = 40, data = 0x8d40cf8 "\002"}, prefix = {len = 35, data = 0x80c0a58 "/usr/local/nginx-1.6.1-nopagespeed/"}, lock_file = {len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}
        ccf = <optimized out>

Another one:

Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Core was generated by `nginx: worker process      '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  ngx_ssl_new_session (ssl_conn=0x8d48c28, sess=0x8d49160) at src/event/ngx_event_openssl.c:1890
1890	    cache = shm_zone->data;
(gdb) bt full
#0  ngx_ssl_new_session (ssl_conn=0x8d48c28, sess=0x8d49160) at src/event/ngx_event_openssl.c:1890
        len = <error reading variable len (Cannot access memory at address 0xbf939be0)>
        p = 0xbf939c98 <error: Cannot access memory at address 0xbf939c98>
        id = <optimized out>
        cached_sess = <optimized out>
        ssl_ctx = <error reading variable ssl_ctx (Cannot access memory at address 0xbf939bf8)>
        shm_zone = 0x0
        c = <error reading variable c (Cannot access memory at address 0xbf939bf0)>
        shpool = <optimized out>
        sess_id = <optimized out>
        cache = <optimized out>
        buf = <error reading variable buf (Cannot access memory at address 0xbf939c0c)>
#1  0xb7637c7c in ?? () from /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0
No symbol table info available.
Cannot access memory at address 0xbf93ac6c

If there's anything else that I can help you tracking the problem, please let me know.

Thank you in advance,
Mike.

[Maxim Dounin]

From the backtrace it looks like duplicate of #235. Please try the workaround outlined there - i.e., configure the same ssl_session_cache for all SNI virtual hosts.