signal 11 on ssllabs test

[typingArtist@…]

Starting a ssllabs.com test to one of my virtual servers does work, i.e. I get the expected results from the test page, however the server thread core dumps twice during the testing process. This is reproducible.

I’m running nginx 1.9.7 with LibreSSL 2.2.4. Both are compiled from sources using Gentoo’s build system. For debugging purposes, nginx is compiled using the debug USE flag and with CFLAGS appended "-ggdb -O0" as well as nostrip FEATURES.

Both core’s created throughout the test indicate the same problem. Looking at line 857 in src/http/ngx_http_request.c it seems that hc->addr->conf->virtual_names is set wrongly.

The respective virtual host is running a rather secure setup (TLS 1.2 and high-quality ciphers only) and has http2 enabled but in other terms it’s standard.

I replaced all sensitive data with mysomething markers.

the important stuff from the config:

listen myipv4:443 ssl http2;
listen [myipv6]:443 ssl http2;
server_name mydomain;

ssl on;
server_tokens off;

ssl_protocols TLSv1.2;
ssl_ciphers 'ECDHE+CHACHA20:ECDHE+AESGCM';

ssl_prefer_server_ciphers on;
ssl_certificate /etc/ssl/nginx/mydomain/ssl.pem;
ssl_certificate_key /etc/ssl/nginx/mydomain/ssl.key;
ssl_ecdh_curve secp384r1;
add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload;";
ssl_buffer_size 1400;
ssl_session_tickets on;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/ssl/nginx/mydomain/full-chain.pem;
resolver mydns valid=300s;
resolver_timeout 15s;

from error_log:

2015/12/01 13:40:06 [notice] 1753#0: using the "epoll" event method
2015/12/01 13:40:06 [notice] 1753#0: nginx/1.9.7
2015/12/01 13:40:06 [notice] 1753#0: OS: Linux 4.3.0-gentoo
2015/12/01 13:40:06 [notice] 1753#0: getrlimit(RLIMIT_NOFILE): 1024:4096
2015/12/01 13:40:06 [notice] 1754#0: start worker processes
2015/12/01 13:40:06 [notice] 1754#0: start worker process 1755
2015/12/01 13:40:06 [notice] 1754#0: start worker process 1756
2015/12/01 13:40:06 [notice] 1754#0: start worker process 1757
2015/12/01 13:40:06 [notice] 1754#0: start worker process 1758
2015/12/01 13:40:06 [notice] 1754#0: start worker process 1760
2015/12/01 13:43:44 [notice] 1754#0: signal 17 (SIGCHLD) received
2015/12/01 13:43:44 [alert] 1754#0: worker process 1756 exited on signal 11 (core dumped)
2015/12/01 13:43:44 [notice] 1754#0: start worker process 1767
2015/12/01 13:43:44 [notice] 1754#0: signal 29 (SIGIO) received

# gdb /usr/sbin/nginx /var/lib/nginx/cores/core.1756
GNU gdb (Gentoo 7.9.1 vanilla) 7.9.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <​http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<​https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<​http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/nginx...done.
[New LWP 1756]

warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `nginx: worker pr'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000462157 in ngx_http_ssl_servername (ssl_conn=0x763a90, ad=0x7ffd8366cfd4, arg=0x0) at src/http/ngx_http_request.c:857
857 src/http/ngx_http_request.c: file not found.
(gdb) backtrace full
#0 0x0000000000462157 in ngx_http_ssl_servername (ssl_conn=0x763a90, ad=0x7ffd8366cfd4, arg=0x0) at src/http/ngx_http_request.c:857

host = {len = 12, data = 0x8327f0 "mydomain"}
servername = 0x836f60 "mydomain"
c = 0x7cae48
hc = 0x830a90
sscf = 0x7fbe4a823048 <ssl_bytes_to_cipher_list+120>
clcf = 0x832240
cscf = 0x7ffd8366cfb4

#1 0x00007fbe4a82e489 in ssl_check_clienthello_tlsext_early () from /usr/lib64/libssl.so.35
No symbol table info available.
#2 0x00007fbe4a816a03 in ssl3_get_client_hello () from /usr/lib64/libssl.so.35
No symbol table info available.
#3 0x00007fbe4a81a725 in ssl3_accept () from /usr/lib64/libssl.so.35
No symbol table info available.
#4 0x00007fbe4a815ad6 in ssl3_read_bytes () from /usr/lib64/libssl.so.35
No symbol table info available.
#5 0x00007fbe4a814151 in ?? () from /usr/lib64/libssl.so.35
No symbol table info available.
#6 0x0000000000446e7e in ngx_ssl_recv (c=0x7cae48, buf=0x76354a "", size=886) at src/event/ngx_event_openssl.c:1343

n = 0
bytes = 0

#7 0x00000000004635bb in ngx_http_read_request_header (r=0x830a90) at src/http/ngx_http_request.c:1380

n = 0
rev = 0x800c00
c = 0x7cae48
cscf = 0x8a

#8 0x0000000000463049 in ngx_http_process_request_headers (rev=0x800c00) at src/http/ngx_http_request.c:1248

p = 0x800b68 "fn\224]Q\001"
len = 0
n = 4294968320
rc = -2
rv = 7505984
h = 0x7557f0
c = 0x7cae48
hh = 0x780878
r = 0x830a90
cscf = 0x27d68
cmcf = 0x757688

#9 0x00000000004362c6 in ngx_event_process_posted (cycle=0x7557f0, posted=0x7292a0 <ngx_posted_events>) at src/event/ngx_event_posted.c:33

q = 0x800c50
ev = 0x800c00

#10 0x0000000000433b77 in ngx_process_events_and_timers (cycle=0x7557f0) at src/event/ngx_event.c:259

flags = 3
timer = 163176
delta = 171

#11 0x00000000004413e5 in ngx_worker_process_cycle (cycle=0x7557f0, data=0x1) at src/os/unix/ngx_process_cycle.c:753

worker = 1

#12 0x000000000043db61 in ngx_spawn_process (cycle=0x7557f0, proc=0x4412f0 <ngx_worker_process_cycle>, data=0x1, name=0x4ec58b "worker process", respawn=-3)

at src/os/unix/ngx_process.c:198

on = 1
pid = 0
s = 1

#13 0x00000000004401d3 in ngx_start_worker_processes (cycle=0x7557f0, n=5, type=-3) at src/os/unix/ngx_process_cycle.c:358

i = 1
ch = {command = 1, pid = 1755, slot = 0, fd = 3}

#14 0x000000000043f7d3 in ngx_master_process_cycle (cycle=0x7557f0) at src/os/unix/ngx_process_cycle.c:130

title = 0x7c7f74 "master process /usr/sbin/nginx"
p = 0x7c7f92 ""
size = 31
i = 1
n = 14

---Type <return> to continue, or q <return> to quit---q
Quit
(gdb) print hc->addr_conf
$1 = (ngx_http_addr_conf_t *) 0x50545448
(gdb) print hc->addr_conf->virtual_names
Cannot access memory at address 0x50545450

[Sergey Kandaurov]

I'll look at it.

[Sergey Kandaurov]

Thank you for the report. Please try the following patch:

diff -r bec5b3093337 src/http/ngx_http_request.c
--- a/src/http/ngx_http_request.c	Tue Nov 17 19:41:39 2015 +0300
+++ b/src/http/ngx_http_request.c	Wed Dec 02 13:11:33 2015 +0300
@@ -837,6 +837,10 @@ ngx_http_ssl_servername(ngx_ssl_conn_t *
 
     c = ngx_ssl_get_connection(ssl_conn);
 
+    if (c->ssl->renegotiation) {
+        return SSL_TLSEXT_ERR_NOACK;
+    }
+
     ngx_log_debug1(NGX_LOG_DEBUG_HTTP, c->log, 0,
                    "SSL server name: \"%s\"", servername);
 

[typingArtist@…]

Whoa, that was fast!

Great, with your patch the worker process doesn’t SIGSEV anymore when performing an ssllabs.com server test. No sign of trouble in error_log or system journal.

Please, close ticket (and apply patch to next release).

[Sergey Kandaurov <pluknet@…>]

In 6320:a6902a941279/nginx:

SSL: fixed possible segfault on renegotiation (ticket #845).

Skip SSL_CTX_set_tlsext_servername_callback in case of renegotiation.
Do nothing in SNI callback as in this case it will be supplied with
request in c->data which isn't expected and doesn't work this way.

This was broken by b40af2fd1c16 (1.9.6) with OpenSSL master branch and LibreSSL.