Segfault at src/core/ngx_palloc.c

[thefoolwhocodes@…]

Running the code with http2 enabled causes frequent segfaults at one of our customer's end.

Backtrace:
#0 0x0000000000415e75 in ngx_log_error_core (level=8, log=0x54ffa50, err=0, fmt=0x543774 "free: %p, unused: %uz") at src/core/ngx_log.c:183

args = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0x7fff19765a50, reg_save_area = 0x7fff19765990}}
p = 0x7fff19765193 ""
last = 0x7fff19765940 ""
msg = 0x7fff1976516f "free: 000000000D008750, unused: 768\n"
n = 11
wrote_stderr = 0
debug_connection = 0
errstr = "2021/06/08 21:10:23 [debug] 16462#0: *11690801 free: 000000000D008750, unused: 768\n", '\000' <repeats 1557 times>...

#1 0x0000000000417243 in ngx_destroy_pool (pool=0xd008750) at src/core/ngx_palloc.c:73

p = 0xd008750
n = 0x0
l = 0x0
c = 0x0

#2 0x000000000044a101 in ngx_memcache_any_read_handler (rev=0x7f12900d83d0) at src/core/ngx_memcache.c:1694

c = 0x7f1290499920
ctx = 0xdfa9c0
available = 3116
consumed = 12
n = 12
readbuffer = 0xdfb180
wq_head = 0xdfa9c8
wq_entry = 0xd008800
rc = 0
reclaim = 1

#3 0x000000000046474f in ngx_epoll_process_events (cycle=0xdf66a0, timer=214, flags=1) at src/event/modules/ngx_epoll_module.c:901

events = 1
revents = 5
instance = 0
i = 0
level = 4544819
err = 0
rev = 0x7f12900d83d0
wev = 0x7f128fd29610
queue = 0x799f20 <ngx_event_timer_sentinel>
c = 0x7f1290499920

#4 0x000000000045234b in ngx_process_events_and_timers (cycle=0xdf66a0) at src/event/ngx_event.c:247

flags = 1
timer = 214
delta = 15101842966

#5 0x0000000000461e63 in ngx_worker_process_cycle (cycle=0xdf66a0, data=0x0) at src/os/unix/ngx_process_cycle.c:719

worker = 0

#6 0x000000000045e8db in ngx_spawn_process (cycle=0xdf66a0, proc=0x461d7c <ngx_worker_process_cycle>, data=0x0, name=0x549bb2 "worker process", respawn=-3) at src/os/unix/ngx_process.c:199

on = 1
pid = 0
s = 0

#7 0x0000000000460de8 in ngx_start_worker_processes (cycle=0xdf66a0, n=8, type=-3) at src/os/unix/ngx_process_cycle.c:344

i = 0

Following commit solves the problem:
​https://github.com/Zimbra/nginx/commit/c46c71698fc6f596e14d1b6309bf6d642f2ab0ef

[Maxim Dounin]

The src/core/ngx_memcache.c file as seen in the backtrace is not something present in nginx. Rather, this looks like a problem in Zimra's modifications, not in nginx itself. Given that removing logging helps, most likely reason is that the log used in the pool is already destroyed when the ngx_destroy_pool() call happens.

Looking into Zimbra code suggests that the log is provided externally in the ngx_memcache_post() call. Most likely, some of the callers fail to keep their logs open till the posted queue item is processed and its pool destroyed. This needs to be found and fixed in the Zimbra code.

Just in case, a "quick fix" solution would be to switch logging to something persistent, like ngx_cycle's log.

Hope this helps.