Opened 3 months ago

Closed 2 months ago

#2586 closed defect (fixed)

probably memory corruption and worker exiting with SIGABRT

Reported by: Georgisim@… Owned by:
Priority: major Milestone: nginx-1.25
Component: documentation Version: 1.25.x
Keywords: Cc:
uname -a: Linux c-d050-u2653-42 5.15.0-40-lowlatency #43-Ubuntu SMP PREEMPT Thu Jun 16 17:07:13 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.25.4
built by gcc 11.2.0 (Ubuntu 11.2.0-19ubuntu1)
built with OpenSSL 3.0.2 15 Mar 2022
TLS SNI support enabled
configure arguments: --prefix=/cdn/nginx_quic --with-cc-opt='-O0 -g -ggdb -march=core2' --with-debug --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --with-ipv6 --with-http_geoip_module --with-http_realip_module --with-http_ssl_module --without-http_charset_module --without-http_ssi_module --without-http_userid_module --without-http_autoindex_module --without-http_scgi_module --without-http_uwsgi_module --without-http_fastcgi_module --without-http_limit_conn_module --without-http_split_clients_module --without-http_limit_req_module --with-http_stub_status_module --with-http_v2_module --with-http_v3_module --with-http_slice_module --with-stream_ssl_module


#0 pthread_kill_implementation (no_tid=0, signo=6, threadid=140309747627840) at ./nptl/pthread_kill.c:44
44 ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
pthread_kill_implementation (no_tid=0, signo=6, threadid=140309747627840) at ./nptl/pthread_kill.c:44
#1 pthread_kill_internal (signo=6, threadid=140309747627840) at ./nptl/pthread_kill.c:78
GI_pthread_kill (threadid=140309747627840, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x00007f9c68aec476 in
GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4 0x00007f9c68ad27f3 in GI_abort () at ./stdlib/abort.c:79
#5 0x00007f9c68b336f6 in
libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f9c68c85b8c "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#6 0x00007f9c68b4ad7c in malloc_printerr (str=str@entry=0x7f9c68c887d0 "double free or corruption (!prev)") at ./malloc/malloc.c:5664
#7 0x00007f9c68b4cefc in _int_free (av=0x7f9c68cc3c80 <main_arena>, p=0x56044e477330, have_lock=<optimized out>) at ./malloc/malloc.c:4591
#8 0x00007f9c68b4f4d3 in GI_libc_free (mem=<optimized out>) at ./malloc/malloc.c:3391
#9 0x000056044c834efc in ngx_destroy_pool (pool=0x56044e477340) at src/core/ngx_palloc.c:90
#10 0x000056044c91fede in ngx_http_v3_close_uni_stream (c=0x7f9c58abc2b0) at src/http/v3/ngx_http_v3_uni.c:102
#11 0x000056044c9205bf in ngx_http_v3_uni_dummy_read_handler (rev=0x7f9c58879450) at src/http/v3/ngx_http_v3_uni.c:273
#12 0x000056044c8a3992 in ngx_quic_close_streams (c=0x7f9c58abf730, qc=0x56044def90e0) at src/event/quic/ngx_event_quic_streams.c:222
#13 0x000056044c8931b7 in ngx_quic_close_connection (c=0x7f9c58abf730, rc=-1) at src/event/quic/ngx_event_quic.c:548
#14 0x000056044c8958c9 in ngx_quic_push_handler (ev=0x56044def96e8) at src/event/quic/ngx_event_quic.c:1432
#15 0x000056044c86d23b in ngx_event_process_posted (cycle=0x56044dba84c0, posted=0x56044c9a3d30 <ngx_posted_events>) at src/event/ngx_event_posted.c:34
#16 0x000056044c86a657 in ngx_process_events_and_timers (cycle=0x56044dba84c0) at src/event/ngx_event.c:263
#17 0x000056044c87b562 in ngx_worker_process_cycle (cycle=0x56044dba84c0, data=0x10) at src/os/unix/ngx_process_cycle.c:721
#18 0x000056044c877a5c in ngx_spawn_process (cycle=0x56044dba84c0, proc=0x56044c87b468 <ngx_worker_process_cycle>, data=0x10, name=0x56044c95cc62 "worker process", respawn=-3) at src/os/unix/ngx_process.c:199
#19 0x000056044c87a2bf in ngx_start_worker_processes (cycle=0x56044dba84c0, n=48, type=-3) at src/os/unix/ngx_process_cycle.c:344
#20 0x000056044c87992e in ngx_master_process_cycle (cycle=0x56044dba84c0) at src/os/unix/ngx_process_cycle.c:130
#21 0x000056044c830ab0 in main (argc=1, argv=0x7ffd9a87a838) at src/core/nginx.c:384
(gdb) q

Here and I can't help a lot for now, seems to be memory corruption when closing the stream and destroying the request pool. I'll try to extract some mode useful info later.

Change History (2)

comment:1 by Roman Arutyunyan <arut@…>, 2 months ago

In 9210:4ed4e1e7f115/nginx:

QUIC: fixed stream cleanup (ticket #2586).

Stream connection cleanup handler ngx_quic_stream_cleanup_handler() calls
ngx_quic_shutdown_stream() after which it resets the pointer from quic stream
to the connection (sc->connection = NULL). Previously if this call failed,
sc->connection retained the old value, while the connection was freed by the
application code. This resulted later in a second attempt to close the freed
connection, which lead to allocator double free error.

The fix is to reset the sc->connection pointer in case of error.

comment:2 by Roman Arutyunyan, 2 months ago

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.