1.4.1 + spdy + centos 6 + openssl-1.0.1e (static), firefox 21 ajax requests ssl spdy = segfault

[Raif Atef]

Hello, on one of my servers, nginx suddenly started crashing on some AJAX-heavy pages when accessed via SSL+SPDY. It seems to happen only when Firefox is the client (tested with Firefox 21), latest version of chrome uses SPDY without crashing.

uname -a:
Linux myserver.com 2.6.32-358.6.2.el6.x86_64 #1 SMP Thu May 16 20:59:36 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

nginx compile flags:
CFLAGS="-g -O0" ./configure --with-pcre=/usr/local/src/nginx-1.4.1/pcre-8.32 --sbin-path=/usr/local/sbin --conf-path=/etc/nginx/nginx.conf --pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --with-http_realip_module --with-http_ssl_module --with-openssl=/usr/local/src/nginx-1.4.1/openssl-1.0.1e --with-http_spdy_module --http-client-body-temp-path=/tmp/nginx_client --http-proxy-temp-path=/tmp/nginx_proxy --http-fastcgi-temp-path=/tmp/nginx_fastcgi --with-http_stub_status_module --with-debug

nginx log when crash happens:
2013/05/19 18:05:58 [notice] 26737#0: start worker process 26899
2013/05/19 18:05:58 [notice] 26737#0: signal 29 (SIGIO) received
2013/05/19 18:05:59 [notice] 26737#0: signal 17 (SIGCHLD) received
2013/05/19 18:05:59 [alert] 26737#0: worker process 26897 exited on signal 11 (core dumped)
2013/05/19 18:05:59 [notice] 26737#0: start worker process 26907
2013/05/19 18:05:59 [notice] 26737#0: signal 29 (SIGIO) received
2013/05/19 18:06:00 [notice] 26737#0: signal 17 (SIGCHLD) received
2013/05/19 18:06:00 [alert] 26737#0: worker process 26899 exited on signal 11 (core dumped)
2013/05/19 18:06:00 [notice] 26737#0: start worker process 26909
2013/05/19 18:06:00 [notice] 26737#0: signal 29 (SIGIO) received

nginx.conf
​http://pastebin.com/G9wAgyeh

gdb backtrace:
# gdb /usr/local/sbin/nginx core.26899

... snip gpl stuff ...

Reading symbols from /usr/local/sbin/nginx...done.
[New Thread 26899]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/50/fc20fea18a6f375789f0f86e28f463d50714fd
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `nginx: worker process '.
Program terminated with signal 11, Segmentation fault.
#0 0x0000003455283c56 in memset_sse2 () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.107.el6.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0 0x0000003455283c56 in memset_sse2 () from /lib64/libc.so.6
#1 0x0000000000493a67 in ngx_http_spdy_state_data (sc=0x3035ba0, pos=0x37c78f8 "", end=0x37c78f8 "")
at src/http/ngx_http_spdy.c:1193
#2 0x0000000000492673 in ngx_http_spdy_state_head (sc=0x3035ba0, pos=0x37c78f8 "", end=0x37c78f8 "")
at src/http/ngx_http_spdy.c:699
#3 0x00000000004919e2 in ngx_http_spdy_read_handler (rev=0x7f0318ffe3b8) at src/http/ngx_http_spdy.c:364
#4 0x000000000042ac31 in ngx_event_process_posted (cycle=0x2893a30, posted=0x8d1b68)
at src/event/ngx_event_posted.c:40
#5 0x000000000042887c in ngx_process_events_and_timers (cycle=0x2893a30) at src/event/ngx_event.c:276
#6 0x0000000000435ebd in ngx_worker_process_cycle (cycle=0x2893a30, data=0x1)
at src/os/unix/ngx_process_cycle.c:807
#7 0x00000000004327ca in ngx_spawn_process (cycle=0x2893a30, proc=0x435cf7 <ngx_worker_process_cycle>,
data=0x1, name=0x609c9b "worker process", respawn=1) at src/os/unix/ngx_process.c:198
#8 0x0000000000435906 in ngx_reap_children (cycle=0x2893a30) at src/os/unix/ngx_process_cycle.c:619
#9 0x00000000004345ed in ngx_master_process_cycle (cycle=0x2893a30) at src/os/unix/ngx_process_cycle.c:180
#10 0x00000000004041b6 in main (argc=3, argv=0x7fffb6c2dbd8) at src/core/nginx.c:412

Server has a Core i3 540 with HT, OS is 64-bit CentOS 6 fully patched (as of date of this message).

• kernel log when error occurred:

May 19 18:06:00 saruman kernel: nginx[26899]: segfault at 0 ip 0000003455283c56 sp 00007fffb6c2d498 error 6 in libc-2.12.so[3455200000+18a000]

The crash is highly reproducible and when it crashes the ip and sp parameters and offsets are always the same.

I hope I've posted enough info.

I maybe a C newbie, but it looks to me that the way firefox 21 is doing spdy causes the request body buffer pointer to be null.

[Valentin V. Bartenev]

The issue is related to the client_body_in_file_only on; setting from your configuration in combination with how Firefox sends empty POST requests. Please, try this patch:

src/http/ngx_http_spdy.c

@@ -1189 +1189 @@
 
         stream->in_closed = 1;
 
-        if (tf) {
+        if (tf && buf) {
             ngx_memzero(buf, sizeof(ngx_buf_t));
 
             buf->in_file = 1;

[Raif Atef]

Attached patch prevents the crash on Firefox 21 with empty POST body AJAX requests. Thank you !
I hadn't noticed this setting in my config, it is set by nginxcp.com custom config, I'll be sure to turn it off (and use the client_body_buffer_size only) now since it seems detrimental to performance.

Thanks again !

[Valentin V. Bartenev]

Finally, a better version of fix was committed as 7542b72fe4b1. Thank you for the report.