#2479 closed defect (fixed)
Quic connection will be closed too early in stream prxoy mode
Reported by: | Owned by: | ||
---|---|---|---|
Priority: | major | Milestone: | |
Component: | other | Version: | 1.21.x |
Keywords: | quic stream busy | Cc: | himac.lee@… |
uname -a: | Linux pekphis107316 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | ||
nginx -V: |
nginx version: nginx/1.23.0
built by gcc 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04) built with OpenSSL 1.1.1 (compatible; BoringSSL) (running with BoringSSL) TLS SNI support enabled configure arguments: --prefix=/usr/local/nginx_http3/nginx --with-debug --with-ld-opt='-Wl,-rpath, -L/work/unitrans90/boringssl/src/build/ssl/ -L//work/boringssl/src/build/crypto' --with-cc-opt='-I//work//boringssl/src/include/ ' --with-stream --with-stream_quic_module --with-http_v2_module --with-http_v3_module --with-http_stub_status_module --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_ssl_module |
Description
When using nginx stream mode, the downstream is QUIC protocol and the stream buffer is set relatively small:
stream {
upstream backend {
server 127.0.0.1:19997;
}
server {
listen 4443 quic reuseport rcvbuf=8M sndbuf=4M;
ssl_alpn h3;
ssl_certificate XX/cert.pem;
ssl_certificate_key XX/secret.key;
ssl_protocols TLSv1.3;
quic_max_udp_payload_size 1472;
quic_stream_buffer_size 64k;
quic_initial_congestion_window 100;
proxy_pass backend;
}
}
if the backend pushes more data, it is likely that the stream buffer will not be enough.
At this time, the ngx_stream_write_filter function will fail, and quic the stream connection is shared, which will cause an error to be returned and the connection will be closed early.
I suggest that ngx_stream_write_filter return eagain instead of error in quic case.
Attachments (1)
Change History (8)
comment:1 by , 19 months ago
comment:2 by , 19 months ago
Thanks for your reply.
The protocol we are proxying with QUIC/Stream is private.
ngx_stream_write_filter failure is caused by send_chain(ngx_quic_stream_send_chain), because the flow control of quic stream is insufficient:
Breakpoint 2, ngx_quic_stream_send_chain (c=0x7f6cab3473d0, in=0x55d0a27fb6f8, limit=0) at src/event/quic/ngx_event_quic_streams.c:938 938 return in; (gdb) bt #0 ngx_quic_stream_send_chain (c=0x7f6cab3473d0, in=0x55d0a27fb6f8, limit=0) at src/event/quic/ngx_event_quic_streams.c:938 #1 0x000055d0a0f2c7e2 in ngx_stream_write_filter (s=<optimized out>, in=<optimized out>, from_upstream=<optimized out>) at src/stream/ngx_stream_write_filter_module.c:261 #2 0x000055d0a0f283ab in ngx_stream_proxy_process (s=s@entry=0x55d0a27fb308, from_upstream=<optimized out>, do_write=<optimized out>) at src/stream/ngx_stream_proxy_module.c:1659 #3 0x000055d0a0f28bce in ngx_stream_proxy_process_connection (ev=<optimized out>, from_upstream=<optimized out>) at src/stream/ngx_stream_proxy_module.c:1510 #4 0x000055d0a0f28c20 in ngx_stream_proxy_upstream_handler (ev=<optimized out>) at src/stream/ngx_stream_proxy_module.c:1400 #5 0x000055d0a0e963b3 in ngx_epoll_process_events (cycle=<optimized out>, timer=<optimized out>, flags=<optimized out>) at src/event/modules/ngx_epoll_module.c:901 #6 0x000055d0a0e8c7d8 in ngx_process_events_and_timers (cycle=cycle@entry=0x55d0a2795600) at src/event/ngx_event.c:248 #7 0x000055d0a0e94518 in ngx_worker_process_cycle (cycle=cycle@entry=0x55d0a2795600, data=data@entry=0x0) at src/os/unix/ngx_process_cycle.c:721 #8 0x000055d0a0e92af6 in ngx_spawn_process (cycle=cycle@entry=0x55d0a2795600, proc=proc@entry=0x55d0a0e9443d <ngx_worker_process_cycle>, data=data@entry=0x0, name=name@entry=0x55d0a10c03db "worker process", respawn=respawn@entry=-4) at src/os/unix/ngx_process.c:199 #9 0x000055d0a0e93cbb in ngx_start_worker_processes (cycle=cycle@entry=0x55d0a2795600, n=1, type=type@entry=-4) at src/os/unix/ngx_process_cycle.c:344 #10 0x000055d0a0e950da in ngx_master_process_cycle (cycle=0x55d0a2795600) at src/os/unix/ngx_process_cycle.c:234 #11 0x000055d0a0e6cc71 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:383 (gdb) n 969 } (gdb) n ngx_stream_write_filter (s=<optimized out>, in=<optimized out>, from_upstream=<optimized out>) at src/stream/ngx_stream_write_filter_module.c:266 266 if (chain == NGX_CHAIN_ERROR) { (gdb) n 271 for (cl = *out; cl && cl != chain; /* void */) { (gdb) n 277 *out = chain; (gdb) 279 if (chain) { (gdb) 280 if (c->shared) { (gdb) 281 ngx_log_error(NGX_LOG_ALERT, c->log, 0, (gdb) 283 return NGX_ERROR; (gdb)
by , 19 months ago
Attachment: | quic-stream-block added |
---|
follow-up: 4 comment:3 by , 19 months ago
Thanks for clarifying this. Indeed, the error is generated for plain UDP here. However, it's not supposed to be generated for QUIC. Please try the attached patch.
comment:4 by , 19 months ago
Replying to Roman Arutyunyan:
Thanks for clarifying this. Indeed, the error is generated for plain UDP here. However, it's not supposed to be generated for QUIC. Please try the attached patch.
Thanks for the patch.
I have test it and work fine.
comment:6 by , 19 months ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
Why do you think ngx_stream_write_filter() will fail? QUIC layer will read as much as it can and then wait for buffer being available later.
Also, could you please share what protocol are you proxying with QUIC/Stream? It was implemented mostly for testing purposes and it's future is still undefined. We'd like to hear more feedback about it.