nginx+ssl crashes very often on low memory environment

[Michael Blue]

Hello,

I'm using nginx version 1.6.1 stable branch and I'm experiencing few hundred crashes/day. First I thought that the problem was caused by ngx_pagespeed, so, I stopped using that. Seems like the bug appeared after I began using ssl certificates, especially in low memory/stress conditions.

openssl version
OpenSSL 1.0.1i 6 Aug 2014

pcre-config --version
8.35

free -m

total used free shared buffers cached

Mem: 1011 937 73 296 11 463
-/+ buffers/cache: 462 548
Swap: 1375 392 983

Traces:

GNU gdb (Debian 7.7.1+dfsg-3) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <​http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i586-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<​http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<​http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/nginx-1.6.1-nopagespeed/sbin/nginx...done.
[New LWP 19233]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Core was generated by `nginx: worker process '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 ngx_ssl_new_session (ssl_conn=0x8fbf3d8, sess=0x8fbb4f8) at src/event/ngx_event_openssl.c:1890
1890 cache = shm_zone->data;
(gdb) bt full
#0 ngx_ssl_new_session (ssl_conn=0x8fbf3d8, sess=0x8fbb4f8) at src/event/ngx_event_openssl.c:1890

len = 140
p = 0xbf939c98 "\001"
id = <optimized out>
cached_sess = <optimized out>
ssl_ctx = 0x8d4ed00
shm_zone = 0x0
c = 0xb051d628
shpool = <optimized out>
sess_id = <optimized out>
cache = <optimized out>
buf = "0\201\211\002\001\001\002\002\003\001\004\002\000\071\004 \211\016\070\031\201\061\253\031M\373\233Wj\303Z\217\b\341\b\n&*)\273\222\267\063\225\313B\217E\004\060\204\365ن\254`\bt\302\305%z\376\230\211\023\256k~\306\362E}g\rG\226@\250*\355\334\377\331*\336\305\362\231\313\360\022]a3\230C\372\241\006\002\004S\373\021U\242\004\002\002\002X\244\006\004\004HTTP\246\022\004\020egcdn.gazduit.ro\001\000\000\000\004", '\000' <repeats 11 times>, "\363\235\223\277\000\000\000 \000\000\000\000 $s\267X\237\223\277\065\211*\267\375\235\223\277@?\374\b@?\374\b\300?\374\b\270\271\373\b\215\237\031_"...

#1 0xb7637c7c in ?? () from /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0
No symbol table info available.
#2 0xb760f684 in ?? () from /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0
No symbol table info available.
#3 0xb763871d in SSL_do_handshake () from /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0
No symbol table info available.
#4 0x0806ef0e in ngx_ssl_handshake (c=0xb051d628) at src/event/ngx_event_openssl.c:785

n = <optimized out>
sslerr = <optimized out>
err = <optimized out>

#5 0x0806f27c in ngx_ssl_handshake_handler (ev=0x8e758b8) at src/event/ngx_event_openssl.c:933

c = 0xb051d628

#6 0x080642a2 in ngx_event_process_posted (cycle=0x8d413b8, posted=0x810694c <ngx_posted_events>) at src/event/ngx_event_posted.c:40

ev = 0x8e758b8

#7 0x08063e83 in ngx_process_events_and_timers (cycle=0x8d413b8) at src/event/ngx_event.c:275

flags = 3
timer = <optimized out>
delta = 443

#8 0x0806a558 in ngx_worker_process_cycle (cycle=0x8d413b8, data=0x1) at src/os/unix/ngx_process_cycle.c:816

worker = <optimized out>
i = <optimized out>
c = <optimized out>

#9 0x08068fa0 in ngx_spawn_process (cycle=0x8d413b8, proc=0x806a493 <ngx_worker_process_cycle>, data=0x1, name=0x80c444a "worker process", respawn=1) at src/os/unix/ngx_process.c:198

on = 1
pid = 0
s = 1

#10 0x0806b06c in ngx_reap_children (cycle=0x8d413b8) at src/os/unix/ngx_process_cycle.c:627

i = <optimized out>
live = 1
n = <optimized out>
ch = {command = 2, pid = 19219, slot = 1, fd = -1}
ccf = <optimized out>

#11 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:180

title = <optimized out>
p = <optimized out>
size = <optimized out>
i = <optimized out>
n = <optimized out>
sigio = 0
set = {val = {0 <repeats 32 times>}}
itv = {it_interval = {tv_sec = 842261476, tv_usec = 170998070}, it_value = {tv_sec = -1217107664, tv_usec = 1}}
live = <optimized out>
delay = 0
ls = <optimized out>

---Type <return> to continue, or q <return> to quit---

ccf = 0x8d41c84

#12 0x0804de3e in main (argc=1, argv=0xbf93b174) at src/core/nginx.c:407

i = <optimized out>
log = <optimized out>
cycle = 0x8d413b8
init_cycle = {conf_ctx = 0x0, pool = 0x8d40cd0, log = 0x8104660 <ngx_log>, new_log = {log_level = 0, file = 0x0, connection = 0, handler = 0x0, data = 0x0, action = 0x0, next = 0x0}, log_use_stderr = 0, files = 0x0,

free_connections = 0x0, free_connection_n = 0, reusable_connections_queue = {prev = 0x0, next = 0x0}, listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, paths = {elts = 0x0, nelts = 0, size = 0, nalloc = 0,

pool = 0x0}, open_files = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0,
pool = 0x0}, connection_n = 0, files_n = 0, connections = 0x0, read_events = 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {len = 50, data = 0x8d40cf8 "\002"}, conf_param = {len = 0, data = 0x0}, conf_prefix = {
len = 40, data = 0x8d40cf8 "\002"}, prefix = {len = 35, data = 0x80c0a58 "/usr/local/nginx-1.6.1-nopagespeed/"}, lock_file = {len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}

ccf = <optimized out>

---

Another one:
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Core was generated by `nginx: worker process '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 ngx_ssl_new_session (ssl_conn=0x8d48c28, sess=0x8d49160) at src/event/ngx_event_openssl.c:1890
1890 cache = shm_zone->data;
(gdb) bt full
#0 ngx_ssl_new_session (ssl_conn=0x8d48c28, sess=0x8d49160) at src/event/ngx_event_openssl.c:1890

len = <error reading variable len (Cannot access memory at address 0xbf939be0)>
p = 0xbf939c98 <error: Cannot access memory at address 0xbf939c98>
id = <optimized out>
cached_sess = <optimized out>
ssl_ctx = <error reading variable ssl_ctx (Cannot access memory at address 0xbf939bf8)>
shm_zone = 0x0
c = <error reading variable c (Cannot access memory at address 0xbf939bf0)>
shpool = <optimized out>
sess_id = <optimized out>
cache = <optimized out>
buf = <error reading variable buf (Cannot access memory at address 0xbf939c0c)>

#1 0xb7637c7c in ?? () from /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0
No symbol table info available.
Cannot access memory at address 0xbf93ac6c

If there's anything else that I can help you tracking the problem, please let me know.

Thank you in advance,
Mike.