Quic connection will be closed too early in stream prxoy mode

[himac.lee@…]

When using nginx stream mode, the downstream is QUIC protocol and the stream buffer is set relatively small:

stream {

upstream backend {

server 127.0.0.1:19997;

}
server {

listen 4443 quic reuseport rcvbuf=8M sndbuf=4M;
ssl_alpn h3;
ssl_certificate XX/cert.pem;
ssl_certificate_key XX/secret.key;
ssl_protocols TLSv1.3;
quic_max_udp_payload_size 1472;
quic_stream_buffer_size 64k;
quic_initial_congestion_window 100;
proxy_pass backend;

}

}

if the backend pushes more data, it is likely that the stream buffer will not be enough.
At this time, the ngx_stream_write_filter function will fail, and quic the stream connection is shared, which will cause an error to be returned and the connection will be closed early.
I suggest that ngx_stream_write_filter return eagain instead of error in quic case.

[Roman Arutyunyan]

Why do you think ngx_stream_write_filter() will fail? QUIC layer will read as much as it can and then wait for buffer being available later.

Also, could you please share what protocol are you proxying with QUIC/Stream? It was implemented mostly for testing purposes and it's future is still undefined. We'd like to hear more feedback about it.

[himac.lee@…]

Thanks for your reply.

The protocol we are proxying with QUIC/Stream is private.

ngx_stream_write_filter failure is caused by send_chain(ngx_quic_stream_send_chain), because the flow control of quic stream is insufficient:

Breakpoint 2, ngx_quic_stream_send_chain (c=0x7f6cab3473d0, in=0x55d0a27fb6f8, limit=0) at src/event/quic/ngx_event_quic_streams.c:938
938	        return in;
(gdb) bt
#0  ngx_quic_stream_send_chain (c=0x7f6cab3473d0, in=0x55d0a27fb6f8, limit=0) at src/event/quic/ngx_event_quic_streams.c:938
#1  0x000055d0a0f2c7e2 in ngx_stream_write_filter (s=<optimized out>, in=<optimized out>, from_upstream=<optimized out>) at src/stream/ngx_stream_write_filter_module.c:261
#2  0x000055d0a0f283ab in ngx_stream_proxy_process (s=s@entry=0x55d0a27fb308, from_upstream=<optimized out>, do_write=<optimized out>) at src/stream/ngx_stream_proxy_module.c:1659
#3  0x000055d0a0f28bce in ngx_stream_proxy_process_connection (ev=<optimized out>, from_upstream=<optimized out>) at src/stream/ngx_stream_proxy_module.c:1510
#4  0x000055d0a0f28c20 in ngx_stream_proxy_upstream_handler (ev=<optimized out>) at src/stream/ngx_stream_proxy_module.c:1400
#5  0x000055d0a0e963b3 in ngx_epoll_process_events (cycle=<optimized out>, timer=<optimized out>, flags=<optimized out>) at src/event/modules/ngx_epoll_module.c:901
#6  0x000055d0a0e8c7d8 in ngx_process_events_and_timers (cycle=cycle@entry=0x55d0a2795600) at src/event/ngx_event.c:248
#7  0x000055d0a0e94518 in ngx_worker_process_cycle (cycle=cycle@entry=0x55d0a2795600, data=data@entry=0x0) at src/os/unix/ngx_process_cycle.c:721
#8  0x000055d0a0e92af6 in ngx_spawn_process (cycle=cycle@entry=0x55d0a2795600, proc=proc@entry=0x55d0a0e9443d <ngx_worker_process_cycle>, data=data@entry=0x0, name=name@entry=0x55d0a10c03db "worker process", respawn=respawn@entry=-4)
    at src/os/unix/ngx_process.c:199
#9  0x000055d0a0e93cbb in ngx_start_worker_processes (cycle=cycle@entry=0x55d0a2795600, n=1, type=type@entry=-4) at src/os/unix/ngx_process_cycle.c:344
#10 0x000055d0a0e950da in ngx_master_process_cycle (cycle=0x55d0a2795600) at src/os/unix/ngx_process_cycle.c:234
#11 0x000055d0a0e6cc71 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:383
(gdb) n
969	}
(gdb) n
ngx_stream_write_filter (s=<optimized out>, in=<optimized out>, from_upstream=<optimized out>) at src/stream/ngx_stream_write_filter_module.c:266
266	    if (chain == NGX_CHAIN_ERROR) {
(gdb) n
271	    for (cl = *out; cl && cl != chain; /* void */) {
(gdb) n
277	    *out = chain;
(gdb) 
279	    if (chain) {
(gdb) 
280	        if (c->shared) {
(gdb) 
281	            ngx_log_error(NGX_LOG_ALERT, c->log, 0,
(gdb) 
283	            return NGX_ERROR;
(gdb)

[Roman Arutyunyan]

Thanks for clarifying this. Indeed, the error is generated for plain UDP here. However, it's not supposed to be generated for QUIC. Please try the attached patch.

[himac.lee@…]

Replying to Roman Arutyunyan:

Thanks for clarifying this. Indeed, the error is generated for plain UDP here. However, it's not supposed to be generated for QUIC. Please try the attached patch.

Thanks for the patch.

I have test it and work fine.

[Roman Arutyunyan]

Committed: ​https://hg.nginx.org/nginx-quic/rev/9ea62b6250f2

[Roman Arutyunyan <arut@…>]

In 9088:9ea62b6250f2/nginx:

Stream: allow waiting on a blocked QUIC stream (ticket #2479).

Previously, waiting on a shared connection was not allowed, because the only
type of such connection was plain UDP. However, QUIC stream connections are
also shared since they share socket descriptor with the listen connection.
Meanwhile, it's perfectly normal to wait on such connections.

The issue manifested itself with stream write errors when the amount of data
exceeded stream buffer size or flow control. Now no error is triggered
and Stream write module is allowed to wait for buffer space to become available.