Opened 5 weeks ago

Closed 7 days ago

#2652 closed defect (invalid)

Some QUIC connections lost domain header in nginx H3

Reported by: bhzhu203@… Owned by:
Priority: major Milestone:
Component: http/3 Version: 1.25.x
Keywords: Cc:
uname -a: Linux jp-proxy 6.5.0-uksm+ #4 SMP PREEMPT_DYNAMIC Tue Aug 29 17:14:48 CST 2023 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.27.0
built by gcc 10.3.1 20211027 (Alpine 10.3.1_git20211027)
built with LibreSSL 3.9.2
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx --with-debug --with-http_v2_module --with-http_v3_module --with-http_realip_module --add-module=../ngx_http_geoip2_module --add-module=../ngx_brotli --with-http_sub_module --with-file-aio --with-threads --with-cc-opt=-I../libressl-3.9.2/build/include --with-ld-opt=-L../libressl-3.9.2/build/lib --with-openssl=../libressl-3.9.

Description

log_format compression escape=json '{"@timestamp":"$time_iso8601",'

'"ip":"$remote_addr","host":"$http_host",'
'"rq":"$request","rqb":"$request_body",'
'"st":"$status","size":$body_bytes_sent,'
'"ua":"$http_user_agent","ck":"$http_cookie",'
'"cost":"$request_time",'
'"ref":"$http_referer",'
'"xff":"$http_x_forwarded_for",'
'"ust":"$upstream_status",'
'"uip":"$upstream_addr",'
'"utm":"$http_utm",'
'"Client-Info":"$http_Client-Info",'
'"timeZone":"$http_timeZone",'
'"countryCode":"$http_countryCode",'
'"useCurrencyCode":"$http_useCurrencyCode",'
'"userId":"$http_userId",'
'"network":"$http_network",'
'"language":"$http_language",'
'"traceId":"$http_traceId",'
'"host1":"$host",'
'"ut":"$upstream_response_time"}';

tail -f /usr/local/nginx/logs/access.log | jq 'select(.host == "" or .host == null)'
{

"@timestamp": "2024-06-14T09:48:54+08:00",
"ip": "115.205.41.187",
"host": "",
"rq": "GET /_nuxt/ebe4dda.js HTTP/3.0",
"rqb": "",
"st": "301",
"size": 162,
"ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
"ck": "",
"cost": "0.182",
"ref": "",
"xff": "",
"ust": "301",
"uip": "127.0.0.1:7377",
"utm": "",
"Client-Info": "-Info",
"timeZone": "",
"countryCode": "",
"useCurrencyCode": "",
"userId": "",
"network": "",
"language": "",
"traceId": "",
"host1": "m.yxxxxxxx.com",
"ut": "0.181"

}
{

"@timestamp": "2024-06-14T09:48:56+08:00",
"ip": "115.205.41.187",
"host": "",
"rq": "GET /_nuxt/ebe4dda.js HTTP/3.0",
"rqb": "",
"st": "301",
"size": 162,
"ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
"ck": "",
"cost": "0.181",
"ref": "",
"xff": "",
"ust": "301",
"uip": "127.0.0.1:7377",
"utm": "",
"Client-Info": "-Info",
"timeZone": "",
"countryCode": "",
"useCurrencyCode": "",
"userId": "",
"network": "",
"language": "",
"traceId": "",
"host1": "m.yxxxxxxx.com",
"ut": "0.181"

}
{

"@timestamp": "2024-06-14T09:52:46+08:00",
"ip": "125.121.8.200",
"host": "",
"rq": "GET /favicon.ico HTTP/3.0",
"rqb": "",
"st": "404",
"size": 110,
"ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
"ck": "",
"cost": "0.190",
"ref": "https://api.yxxxxxxx.com/",
"xff": "",
"ust": "404",
"uip": "127.0.0.1:7377",
"utm": "",
"Client-Info": "-Info",
"timeZone": "",
"countryCode": "",
"useCurrencyCode": "",
"userId": "",
"network": "",
"language": "",
"traceId": "",
"host1": "api.yxxxxxxx.com",
"ut": "0.190"

}

When the QUIC protocol is uncommented for just one domain (api.yxxxxxxx.com):

### 1. H3 connections may lose the $http_host variable (it becomes null), but the $host variable is still present.

### 2. If H3 is uncommented for api.yxxxxxxx.com (just this one domain), other domains that don't have H3 uncommented will still accept H3 connections. This will mess up incoming connections, especially for domains sharing certificates, causing them to resolve to the wrong domain and result in incorrect 301 redirects.

https://github.com/koverstreet/bcachefs/assets/3523160/998feb4f-314b-4977-a8fe-bc0a1babbccd

### 3. Some connections get stuck in an endless loop of 301 redirects.

Change History (10)

comment:2 by bhzhu203@…, 5 weeks ago

I conducted some tests and found that:

If the domains are not in the server lists, they will redirect to the first QUIC server instead of the default QUIC server. The "server_name _;" directive does not help. It must be placed in the first server block to take effect.

There is no issue if the domain is in the server_name list.

However, the issue with the missing $http_host header/variable still exists.

comment:3 by Roman Arutyunyan, 4 weeks ago

See #2659 for $http_host issue.

comment:4 by Roman Arutyunyan, 4 weeks ago

Could you please elaborate on the default server issue. In the following example the last server acts as a default and handles qux.example.com.

    server {
        listen 9443 quic reuseport;
        server_name example.com;
        ...
    }


    server {
        listen 9443 quic;
        server_name foo.example.com;
        ...
    }

    server {
        listen 9443 quic default_server;
        server_name _;
        ...
    }

in reply to:  4 comment:5 by bhzhu203@…, 4 weeks ago

Replying to Roman Arutyunyan:

Could you please elaborate on the default server issue. In the following example the last server acts as a default and handles qux.example.com.

    server {
        listen 9443 quic reuseport;
        server_name example.com;
        ...
    }


    server {
        listen 9443 quic;
        server_name foo.example.com;
        ...
    }

    server {
        listen 9443 quic default_server;
        server_name _;
        ...
    }

I have tested :

This is successful

    server {
        listen 443 quic reuseport;
        server_name example.com;
        ...
    }


    server {
        listen 443 quic;
        server_name foo.example.com;
        ...
    }

    server {
        listen 443 quic default_server;
        server_name _;
        ...
    }

This is failed (Because losing the "default_server" ) . Does "server_name _;" equals to default_server ?

    server {
        listen 443 quic reuseport;
        server_name example.com;
        ...
    }


    server {
        listen 443 quic;
        server_name foo.example.com;
        ...
    }

    server {
        listen 443 quic ;
        server_name _;
        ...
    }
Last edited 3 weeks ago by Roman Arutyunyan (previous) (diff)

comment:6 by Roman Arutyunyan, 3 weeks ago

In the last config client will never get into the last server since the server name is (deliberately, but meaninglessly) invalid. Connections to foo.example.com will go to the second server and all other connections will go to the first one. If default_server is not specified, the first server will be default. if you want to reject connections to servers which do not have http/3 explicitly enabled, trigger an error in the default server.

in reply to:  6 comment:7 by bhzhu203@…, 3 weeks ago

Replying to Roman Arutyunyan:

In the last config client will never get into the last server since the server name is (deliberately, but meaninglessly) invalid. Connections to foo.example.com will go to the second server and all other connections will go to the first one. If default_server is not specified, the first server will be default. if you want to reject connections to servers which do not have http/3 explicitly enabled, trigger an error in the default server.

So ,this means that the 'default_server' has the highest priority. However, the server_name block '_' is quite confusing.

comment:8 by Roman Arutyunyan, 3 weeks ago

Default server is the server that's chosen for the connection by default. Server name '_' has no other meaning but just a fake name that would not match any real server name. You can read this for details:

https://nginx.org/en/docs/http/server_names.html

in reply to:  8 comment:9 by bhzhu203@…, 10 days ago

Replying to Roman Arutyunyan:

Default server is the server that's chosen for the connection by default. Server name '_' has no other meaning but just a fake name that would not match any real server name. You can read this for details:

https://nginx.org/en/docs/http/server_names.html

I have understood . Thank you very much

comment:10 by Roman Arutyunyan, 7 days ago

Resolution: invalid
Status: newclosed
Note: See TracTickets for help on using tickets.