Opened 4 months ago

Closed 4 months ago

#2206 closed defect (invalid)

Segfault at src/core/ngx_palloc.c

Reported by: thefoolwhocodes@… Owned by:
Priority: minor Milestone:
Component: documentation Version: 1.19.x
Keywords: Cc:
uname -a: Linux shashi-classic 4.15.0-1070-oracle #78~16.04.1-Ubuntu SMP Tue Apr 13 19:58:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.19.0
built with OpenSSL 1.1.1k FIPS 25 Mar 2021 (running with OpenSSL 1.1.1g 21 Apr 2020)
TLS SNI support enabled
configure arguments: --prefix=/opt/zimbra/common --with-cc-opt='-g -I/opt/zimbra/common/include' --with-ld-opt='-Wl,-rpath,/opt/zimbra/common/lib -L/opt/zimbra/common/lib' --with-debug --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_v2_module --with-pcre --with-mail --with-mail-sasl --with-mail_ssl_module --error-log-path=/opt/zimbra/log/nginx.log --http-log-path=/opt/zimbra/log/nginx.access.log --http-client-body-temp-path=/opt/zimbra/data/tmp/nginx/client --http-proxy-temp-path=/opt/zimbra/data/tmp/nginx/proxy --http-fastcgi-temp-path=/opt/zimbra/data/tmp/nginx/fastcgi --without-http_scgi_module --without-http_uwsgi_module --add-module=modules/nviennot-nginx-tcp-keepalive --add-module=zmmodules/http/sso --add-module=zmmodules/http/upstreamzmauth --add-module=zmmodules/mail/zmauth --add-module=zmmodules/mail/throttle

Description

Running the code with http2 enabled causes frequent segfaults at one of our customer's end.

Backtrace:
#0 0x0000000000415e75 in ngx_log_error_core (level=8, log=0x54ffa50, err=0, fmt=0x543774 "free: %p, unused: %uz") at src/core/ngx_log.c:183

args = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0x7fff19765a50, reg_save_area = 0x7fff19765990}}
p = 0x7fff19765193 ""
last = 0x7fff19765940 ""
msg = 0x7fff1976516f "free: 000000000D008750, unused: 768\n"
n = 11
wrote_stderr = 0
debug_connection = 0
errstr = "2021/06/08 21:10:23 [debug] 16462#0: *11690801 free: 000000000D008750, unused: 768\n", '\000' <repeats 1557 times>...

#1 0x0000000000417243 in ngx_destroy_pool (pool=0xd008750) at src/core/ngx_palloc.c:73

p = 0xd008750
n = 0x0
l = 0x0
c = 0x0

#2 0x000000000044a101 in ngx_memcache_any_read_handler (rev=0x7f12900d83d0) at src/core/ngx_memcache.c:1694

c = 0x7f1290499920
ctx = 0xdfa9c0
available = 3116
consumed = 12
n = 12
readbuffer = 0xdfb180
wq_head = 0xdfa9c8
wq_entry = 0xd008800
rc = 0
reclaim = 1

#3 0x000000000046474f in ngx_epoll_process_events (cycle=0xdf66a0, timer=214, flags=1) at src/event/modules/ngx_epoll_module.c:901

events = 1
revents = 5
instance = 0
i = 0
level = 4544819
err = 0
rev = 0x7f12900d83d0
wev = 0x7f128fd29610
queue = 0x799f20 <ngx_event_timer_sentinel>
c = 0x7f1290499920

#4 0x000000000045234b in ngx_process_events_and_timers (cycle=0xdf66a0) at src/event/ngx_event.c:247

flags = 1
timer = 214
delta = 15101842966

#5 0x0000000000461e63 in ngx_worker_process_cycle (cycle=0xdf66a0, data=0x0) at src/os/unix/ngx_process_cycle.c:719

worker = 0

#6 0x000000000045e8db in ngx_spawn_process (cycle=0xdf66a0, proc=0x461d7c <ngx_worker_process_cycle>, data=0x0, name=0x549bb2 "worker process", respawn=-3) at src/os/unix/ngx_process.c:199

on = 1
pid = 0
s = 0

#7 0x0000000000460de8 in ngx_start_worker_processes (cycle=0xdf66a0, n=8, type=-3) at src/os/unix/ngx_process_cycle.c:344

i = 0

Following commit solves the problem:
https://github.com/Zimbra/nginx/commit/c46c71698fc6f596e14d1b6309bf6d642f2ab0ef

Change History (1)

comment:1 by Maxim Dounin, 4 months ago

Resolution: invalid
Status: newclosed

The src/core/ngx_memcache.c file as seen in the backtrace is not something present in nginx. Rather, this looks like a problem in Zimra's modifications, not in nginx itself. Given that removing logging helps, most likely reason is that the log used in the pool is already destroyed when the ngx_destroy_pool() call happens.

Looking into Zimbra code suggests that the log is provided externally in the ngx_memcache_post() call. Most likely, some of the callers fail to keep their logs open till the posted queue item is processed and its pool destroyed. This needs to be found and fixed in the Zimbra code.

Just in case, a "quick fix" solution would be to switch logging to something persistent, like ngx_cycle's log.

Hope this helps.

Note: See TracTickets for help on using tickets.