segfault in quic

[Georgisim@…]

warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing
[New LWP 138846]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `nginx: worker process '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fcd4014a214 in EVP_CIPHER_CTX_is_encrypting () from /lib/x86_64-linux-gnu/libcrypto.so.3
(gdb) bt
#0 0x00007fcd4014a214 in EVP_CIPHER_CTX_is_encrypting () from /lib/x86_64-linux-gnu/libcrypto.so.3
#1 0x00005646ac29ff3f in ngx_quic_crypto_common (s=0x5646adeb3470, out=0x7ffd2593ad28, nonce=0x7ffd2593ab4c "", in=0x7ffd2593ab20, ad=0x7ffd2593ab30, log=0x5646af2d8100)

at src/event/quic/ngx_event_quic_protection.c:493

#2 0x00005646ac29fea3 in ngx_quic_crypto_open (s=0x5646adeb3470, out=0x7ffd2593ad28, nonce=0x7ffd2593ab4c "", in=0x7ffd2593ab20, ad=0x7ffd2593ab30, log=0x5646af2d8100)

at src/event/quic/ngx_event_quic_protection.c:458

#3 0x00005646ac2a1f09 in ngx_quic_decrypt (pkt=0x7ffd2593ac60, largest_pn=0x5646ae310a58) at src/event/quic/ngx_event_quic_protection.c:1190
#4 0x00005646ac298727 in ngx_quic_handle_payload (c=0x7fcd2fd48790, pkt=0x7ffd2593ac60) at src/event/quic/ngx_event_quic.c:982
#5 0x00005646ac29801f in ngx_quic_handle_packet (c=0x7fcd2fd48790, conf=0x0, pkt=0x7ffd2593ac60) at src/event/quic/ngx_event_quic.c:843
#6 0x00005646ac2979ab in ngx_quic_handle_datagram (c=0x7fcd2fd48790, b=0x7ffd2593ae90, conf=0x0) at src/event/quic/ngx_event_quic.c:693
#7 0x00005646ac296d31 in ngx_quic_input_handler (rev=0x7fcd2fb07310) at src/event/quic/ngx_event_quic.c:436
#8 0x00005646ac29a093 in ngx_quic_recvmsg (ev=0x7fcd2fb07070) at src/event/quic/ngx_event_quic_udp.c:195
#9 0x00005646ac2821f2 in ngx_epoll_process_events (cycle=0x5646add304c0, timer=98, flags=1) at src/event/modules/ngx_epoll_module.c:901
#10 0x00005646ac26e5b7 in ngx_process_events_and_timers (cycle=0x5646add304c0) at src/event/ngx_event.c:248
#11 0x00005646ac27f562 in ngx_worker_process_cycle (cycle=0x5646add304c0, data=0x1a) at src/os/unix/ngx_process_cycle.c:721
#12 0x00005646ac27ba5c in ngx_spawn_process (cycle=0x5646add304c0, proc=0x5646ac27f468 <ngx_worker_process_cycle>, data=0x1a, name=0x5646ac360c62 "worker process", respawn=-3) at src/os/unix/ngx_process.c:199
#13 0x00005646ac27e2bf in ngx_start_worker_processes (cycle=0x5646add304c0, n=48, type=-3) at src/os/unix/ngx_process_cycle.c:344
#14 0x00005646ac27d92e in ngx_master_process_cycle (cycle=0x5646add304c0) at src/os/unix/ngx_process_cycle.c:130
#15 0x00005646ac234ab0 in main (argc=1, argv=0x7ffd2593b718) at src/core/nginx.c:384
(gdb) d 1
No breakpoint number 1.
(gdb) f 1
#1 0x00005646ac29ff3f in ngx_quic_crypto_common (s=0x5646adeb3470, out=0x7ffd2593ad28, nonce=0x7ffd2593ab4c "", in=0x7ffd2593ab20, ad=0x7ffd2593ab30, log=0x5646af2d8100)

at src/event/quic/ngx_event_quic_protection.c:493

warning: Source file is more recent than executable.
493 enc = EVP_CIPHER_CTX_encrypting(ctx);
(gdb) p ctx
$1 = (EVP_CIPHER_CTX *) 0x0
(gdb) f 2
#2 0x00005646ac29fea3 in ngx_quic_crypto_open (s=0x5646adeb3470, out=0x7ffd2593ad28, nonce=0x7ffd2593ab4c "", in=0x7ffd2593ab20, ad=0x7ffd2593ab30, log=0x5646af2d8100)

at src/event/quic/ngx_event_quic_protection.c:458

458 return ngx_quic_crypto_common(s, out, nonce, in, ad, log);
(gdb) p s
$2 = (ngx_quic_secret_t *) 0x5646adeb3470
(gdb) p *s
$3 = {secret = {len = 0, data = '\000' <repeats 47 times>}, iv = {len = 0, data = '\000' <repeats 11 times>}, hp = {len = 0, data = '\000' <repeats 47 times>}, ctx = 0x0, hp_ctx = 0x0}
(gdb) f 3
#3 0x00005646ac2a1f09 in ngx_quic_decrypt (pkt=0x7ffd2593ac60, largest_pn=0x5646ae310a58) at src/event/quic/ngx_event_quic_protection.c:1190
1190 if (ngx_quic_crypto_open(secret, &pkt->payload, nonce, &in, &ad, pkt->log)
(gdb) p secret->ctx
$4 = (EVP_CIPHER_CTX *) 0x0
(gdb) p *secret
$5 = {secret = {len = 0, data = '\000' <repeats 47 times>}, iv = {len = 0, data = '\000' <repeats 11 times>}, hp = {len = 0, data = '\000' <repeats 47 times>}, ctx = 0x0, hp_ctx = 0x0}
(gdb)
(gdb) p *secret
$5 = {secret = {len = 0, data = '\000' <repeats 47 times>}, iv = {len = 0, data = '\000' <repeats 11 times>}, hp = {len = 0, data = '\000' <repeats 47 times>}, ctx = 0x0, hp_ctx = 0x0}
(gdb) p pkt->keys->secrets[pkt->level].client
$6 = {secret = {len = 32, data = "\253\354ѫV\006\220\355Qa\221\334\324\377>=\027\273\016z\177\277Q4\025\n\225\026a\262\066\263", '\000' <repeats 15 times>}, iv = {len = 12,

data = "\224h5\025C\265\066\357E\253rZ"}, hp = {len = 16, data = "&P\314ԏ혎X\275\071\222\300\356Ӓ", '\000' <repeats 31 times>}, ctx = 0x5646adea0970, hp_ctx = 0x5646af284220}

(gdb) p pkt->keys->secrets[pkt->level].client->ctx
$7 = (EVP_CIPHER_CTX *) 0x5646adea0970
(gdb) p pkt->keys->next_key.client
$9 = {secret = {len = 0, data = '\000' <repeats 47 times>}, iv = {len = 0, data = '\000' <repeats 11 times>}, hp = {len = 0, data = '\000' <repeats 47 times>}, ctx = 0x0, hp_ctx = 0x0}
(gdb) p/t pkt->flags
$12 = 1111101
(gdb)
(gdb) p key_phase
$13 = 1
(gdb) p pkt->key_phase
$14 = 0
(gdb)

Looking at the backtrace, I saw problem appears in ngx_quic_crypto_common trying to call EVP_CIPHER_CTX_encrypting(ctx) with ctx pointing to NULL.
Tracing back through ngx_quic_crypto_common(), ngx_quic_crypto_open(), way don to ngx_quic_decrypt() we can see that this i actually secret->ctx is NULL.
Further we see that secret is taken from pkt->keys->secrets[pkt->level].client (line 1115 in ngx_quic_decrypt()) where ctx is has non zero value = 0x5646adea0970, but on line 1147
it overriden by secret = &pkt->keys->next_key.client; which has cts NULL. So it seems that key is about to be updated, but pkt->keys->next_key is to zero initialized structure.

I see that this could happen in ngx_quic_keys_switch() and ngx_quic_keys_cleanup(), so probably some wrong sequence makes worker to crash. I'm not yet so familiar yet with quic protocol
and its nginx implementation.

[Georgisim@…]

example configuration identical to real one, where the problem appeared

[Roman Arutyunyan]

Thanks for reporting this. We are looking into this issue now.
If you can post (or send directly to arut@…) the debug log, it would help a lot.

[Georgisim@…]

Yes, I know, but unfortunately I don't have debug log - its on production machine and happened several times. May be we will try with in-memory debug, which helped in the past when we found segfault in SSL module. If you think full coredump will easy the task, I can strip certs and send it to you in private.

[Roman Arutyunyan]

Could you please also post the contents of pkt in ngx_quic_handle_packet().
(gdb) p *pkt

[Georgisim@…]

(gdb) p *pkt
$26 = {log = 0x5646af2d8100, path = 0x5646aee9fdc0, keys = 0x5646adeb2fb0, received = 0, number = 0, num_len = 0 '\000', trunc = 0, flags = 125 '}', version = 0, token = {len = 0, data = 0x0},

level = ssl_encryption_application, error = 0, raw = 0x7ffd2593ae90,
data = 0x5646ac3d4475 <buffer+85> "g\217\315X^{\225\327\256\311ZDO\263\023\337hv1\205\362F\342\306\362T܀\004>ō\374P\001\354\067\026F:\233$\031.\005ԑFg9\016\301\374\031 \273\274\350)!(\361\062r\327\070\223\216\315\331\332\345R\332\003\321\067\v(\320Uy\n\303\034⨳\031\025)", len = 1115, odcid = {len = 0, data = 0x0}, odcid_buf = '\000' <repeats 19 times>, dcid = {len = 20,}

data = 0x5646ac3d4476 <buffer+86> "\217\315X^{\225\327\256\311ZDO\263\023\337hv1\205\362F\342\306\362T܀\004>ō\374P\001\354\067\026F:\233$\031.\005ԑFg9\016\301\374\031 \273\274\350)!(\361\062r\327\070\223\216\315\331\332\345R\332\003\321\067\v(\320Uy\n\303\034⨳\031\025)"}, scid = {len = 0, data = 0x0}, pn = 6988,}

plaintext = 0x5646ac3c4420 <buf> "}\217\315X^{\225\327\256\311ZDO\263\023\337hv1\205\362F\033L\031)<", payload = {len = 1076, data = 0x5646ac3c4437 <buf+23> "\031)<"}, need_ack = 0, key_phase = 0,
key_update = 1, parsed = 1, decrypted = 0, validated = 0, retried = 0, first = 0, rebound = 0, path_challenged = 0}}

[Sergey Kandaurov <pluknet@…>]

In 9209:1bf1b423f268/nginx:

QUIC: trial packet decryption in response to invalid key update.

Inspired by RFC 9001, Section 6.3, trial packet decryption with the current
keys is now used to avoid a timing side-channel signal. Further, this fixes
segfault while accessing missing next keys (ticket #2585).

[m.herasimovich]

Milestone renamed