When reuseport is not present with listen 443 quic, only HTTP/2 works not HTTP/3

[skygunner@…]

listen 443 quic;
If reuseport is not present with the above listen line, then only HTTP/2 works not HTTP/3. Add reuseport then it works fine. (chrome/firefox).

debug log:

2023/04/14 17:16:24 [debug] 188209#188209: *1 free: 0000AAAAFFA50830, unused: 2888
2023/04/14 17:16:24 [debug] 188209#188209: *1 free: 0000AAAAFFAB8130
2023/04/14 17:16:24 [debug] 188209#188209: quic recvmsg on 0.0.0.0:443, ready: 1
2023/04/14 17:16:24 [debug] 188209#188209: posix_memalign: 0000AAAAFF9D3E00:512 @16
2023/04/14 17:16:24 [debug] 188209#188209: malloc: 0000AAAAFFA50830:1250
2023/04/14 17:16:24 [debug] 188209#188209: *3 quic recvmsg: 220.233.6.16:13281 fd:10 n:1250
2023/04/14 17:16:24 [debug] 188209#188209: *3 http3 init session
2023/04/14 17:16:24 [debug] 188209#188209: *3 posix_memalign: 0000AAAAFFA50D20:512 @16
2023/04/14 17:16:24 [debug] 188209#188209: *3 add cleanup: 0000AAAAFFA50EE0
2023/04/14 17:16:24 [debug] 188209#188209: *3 event timer add: 10: 180000:11113282
2023/04/14 17:16:24 [debug] 188209#188209: *3 quic run
2023/04/14 17:16:24 [debug] 188209#188209: *3 quic packet rx long flags:c4 version:1
2023/04/14 17:16:24 [debug] 188209#188209: *3 quic packet rx init len:1232
2023/04/14 17:16:24 [debug] 188209#188209: *3 quic packet rx dcid len:8 80be092b143c86cd
2023/04/14 17:16:24 [debug] 188209#188209: *3 quic packet rx scid len:0 
2023/04/14 17:16:24 [debug] 188209#188209: *3 quic address validation token len:0 
2023/04/14 17:16:24 [debug] 188209#188209: *3 sendmsg: 107 of 107
2023/04/14 17:16:24 [debug] 188209#188209: *3 quic retry packet sent to 
2023/04/14 17:16:24 [debug] 188209#188209: *3 quic packet done rc:-4 level:init decr:0 pn:0 perr:0
2023/04/14 17:16:24 [debug] 188209#188209: *3 quic packet rejected rc:-4, cleanup connection
2023/04/14 17:16:24 [debug] 188209#188209: *3 reusable connection: 0
2023/04/14 17:16:24 [debug] 188209#188209: *3 run cleanup: 0000AAAAFFA50EE0
2023/04/14 17:16:24 [debug] 188209#188209: *3 event timer del: -1: 11113282
2023/04/14 17:16:24 [debug] 188209#188209: *3 free: 0000AAAAFFA50830
2023/04/14 17:16:24 [debug] 188209#188209: *3 free: 0000AAAAFF9D3E00, unused: 8
2023/04/14 17:16:24 [debug] 188209#188209: *3 free: 0000AAAAFFA50D20, unused: 40
2023/04/14 17:16:24 [debug] 188209#188209: quic recvmsg() not ready (11: Resource temporarily unavailable)
2023/04/14 17:16:24 [debug] 188209#188209: *1 http2 idle handler
2023/04/14 17:16:24 [debug] 188209#188209: *1 reusable connection: 0
2023/04/14 17:16:24 [debug] 188209#188209: *1 posix_memalign: 0000AAAAFF9CA1E0:4096 @16
2023/04/14 17:16:24 [debug] 188209#188209: *1 http2 read handler
2023/04/14 17:16:24 [debug] 188209#188209: *1 SSL_read: 158
2023/04/14 17:16:24 [debug] 188209#188209: *1 SSL_read: -1
2023/04/14 17:16:24 [debug] 188209#188209: *1 SSL_get_error: 2

[Sergey Kandaurov]

Please provide the full relevant nginx config w.r.t. the listen directives.

[Sergey Kandaurov]

Feedback timeout.

[m.herasimovich]

Milestone renamed

[marlonanjos@…]

Hi there,

I got in the same condition, so I've attached the error.log with debug option in case it help you guys.

uname:

Linux marlon-before 5.15.0-118-generic #128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Nginx -V:

nginx version: nginx/1.26.1
built with OpenSSL 3.0.2 15 Mar 2022
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -ffile-prefix-map=/build/nginx-ufANOL/nginx-1.26.1=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_v3_module --with-http_gzip_static_module --with-http_perl_module --with-threads --with-file-aio --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --without-select_module --without-poll_module --without-http_ssi_module --without-http_userid_module --without-http_mirror_module --without-http_autoindex_module --without-http_split_clients_module --without-http_referer_module --without-http_uwsgi_module --without-http_scgi_module --without-http_grpc_module --without-http_memcached_module --without-http_empty_gif_module --without-http_browser_module --without-http_upstream_hash_module --without-http_upstream_ip_hash_module --without-http_upstream_least_conn_module --without-http_upstream_random_module --without-http_upstream_zone_module

While testing with curl:

curl --http3-only https://marlon-before.tempurl.host/
curl: (56) QUIC connection has been shut down

If I use curl -v I can see that some data is being received before the connection shutdown:

<div class="wp-block-group alignfull has-base-2-background-color has-background has-global-padding is-layout-constrained wp-container-core-group-is-layout-9 wp-block-group-is-layout-constrained" style="margin-top:0;margin-bottom:0;padding-top:var(--wp--preset--spacing--50);padding-right:var(--wp--preset--spacing--50);padding-bottom:var(--wp--preset--spacing--50);padding-left:var(--wp--preset--spacing--50)">
	
* QUIC connection has been shut down
* Connection #0 to host marlon-before.tempurl.host left intact
curl: (56) QUIC connection has been shut down

relevant conf:

http {
...
http3 on;
...
}

server {
  listen      443 ssl;
  listen [::]:443 ssl;
  listen      443 quic;
  listen [::]:443 quic;

...
}

(The same happens with Chrome Version 127.0.6533.119 (Official Build) (64-bit))

[magacola285@…]

The issue can be fixed with :

 listen [::]:443 ssl default_server ipv6only=off backlog=1024;
 listen 443 quic reuseport ipv6only=off; 

Posting my nginx config:

$ nginx -V
nginx version: nginx/1.27.1
built by gcc 11.4.1 20231218 (Red Hat 11.4.1-3) (GCC) 
built with LibreSSL 3.9.2
TLS SNI support enabled
configure arguments: --prefix=/usr/local --sbin-path=/usr/local/sbin/nginx --modules-path=/usr/local/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/log/nginx/error_log --http-log-path=/log/nginx/access_log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-compat --with-debug --with-file-aio --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_perl_module=dynamic --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-pcre --with-pcre-jit --with-stream_ssl_module --with-threads --with-http_v3_module --with-openssl=/home/smart/work/libressl-3.9.2 --without-quic_bpf_module --with-http_perl_module=dynamic --with-libatomic