Unexpected intermittent behavior of map directive(s)

[me.niklasbeierl.io@…]

I have a single Https server configured, it uses one certificate for all the domains.

What I want it to do is this:

app.com/static/* -> Serve from filesystem
app.com -> Proxy to app server
staging.app.com/static/* -> Serve from filesystem (different folder)
staging.app.com -> Proxy to staging app server

Config:

# Use docker DNS to resolve other services
resolver 127.0.0.11 valid=30s;

# Choose upstream based on server name
map $ssl_server_name $targetBackend {
    # volatile; # doesnt fix it
    app.com http://production:81/; 
    staging.app.com http://staging:81/;
}

# Choose static dir based on ssl server
map $ssl_server_name $staticPath {
    # volatile; # doesnt fix it
    myapp.com /var/www/staging-static/;
    staging.myapp.com /var/www/production-static/;
}

server {
     listen 443 ssl http2;
     listen [::]:443 ssl http2;

     server_name app.com;
     server_name staging.app.com;

     ssl_certificate /etc/letsencrypt/live/x/cert.pem;
     ssl_certificate_key /etc/letsencrypt/live/x/privkey.pem;
     ssl_session_timeout 1d;
     ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions
     ssl_session_tickets off;

     ssl_dhparam /etc/nginx/conf.d/dhparam;
     ssl_protocols TLSv1.2 TLSv1.3;
     ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
     ssl_prefer_server_ciphers off;


    # If path is sub-path of /static, serve from some dir
    location /static/ {
        alias $staticPath;
    }
   
    # Forward everything else to app servers
    location / {
        proxy_pass $targetBackend;
    }
}

In my test setup, I have the upstream servers send a retrun page that identifies them: "Hello, I am X". And the two static folders also have a simple index.html identifying them.

It only sometimes behaves as expected. Some weirdness I often get are those:

Open staging.app.com/static/ -> Get staging static
Open app.com/static/ -> Get staging static again ??

Or:

Open staging.app.com/static/ -> Get staging static
Open app.com/static/ -> Get production static
Open app.com/static/ -> Get staging static ???

I have tried adding volatile to the maps but it did not fix the issue.

[Maxim Dounin]

The $ssl_server_name variable does not reflect the host being requested. Rather, it is a ​SNI name sent by the client during SSL handshake. It might not be available at all, and even if available - it might not match the host being requested at the HTTP protocol level.

In particular, HTTP/2 explicitly specifies ​connection reuse rules, which define when connections to different hosts can be reused for requests to other hosts.

Instead of using $ssl_server_name, consider using ​$host, it should fix the configuration.

[me.niklasbeierl.io@…]

Hey there,

yes indeed. $host works correctly and is semantically more in line with what I want. However, there might still be an issue:
A colleague of mine pointed out exactly what you mentioned: My browser might just re-use the connection and not send the different SNI.

So we spun up wireshark and discovered that my browser DOES send the different server name in the Client Hello for both requests, but nginx just ignored it and answered both requests as if they where for staging. :/

I can provide pcap non-publicly if you want.

[Maxim Dounin]

The server name as sent during an abbreviated handshake is typically ignored, so that's might be what you've seen in Wireshark. Try disabling session cache - this might help to see the new server name on the new connection if it's re-established by the browser for the new request.

[me.niklasbeierl.io@…]

I will try to take a deeper look at this tomorrow.

In the meantime I would like to suggest explicitly documenting that $ssl_server_name does not necessarily reflect the hostname in the requested URL. I think this is rather un-intuitive and one could, for example, end up building an authentication bypass:

map $host $staticPath {
    volatile;
    staging.app.com /var/www/secret/;
    app.com /var/www/public/;
}

map $ssl_server_name $auth_by_domain {
    volatile;
    staging.app.com "Staging Area";
    app.com "off";
}

server {

    location / {
        alias $staticPath;
        auth_basic $auth_by_domain;
        auth_basic_user_file /etc/nginx/users;
    }
}

[Maxim Dounin]

To re-iterate: SNI name is not expected to match the hostname requested at the HTTP level, and can be arbitrary or not present at all. I don't really see where the idea to use it instead of the hostname provided at the HTTP level comes from, but obviously enough it is no better than using any arbitrary client-provided string instead, and can result in arbitrary issues.

Note well that the documentation already defines ​$ssl_server_name pretty clearly as "the server name requested through SNI". It also links to the Wikipedia article about SNI, which discusses various aspects of SNI usage, including ​domain fronting, which is specifically about using different domains in SNI and at the HTTP level.

I don't think I've previously seen attempts to use SNI name instead of the hostname requested at the HTTP level, and I don't really think the documentation needs additional clarification. If there will be more such misuses observed, we'll consider updating the documentation to make this more obvious.

[me.niklasbeierl.io@…]

I agree and understand what you say about SNI. I also agree that using $ssl_server_name for my purpose was wrong.

And it also turns out that my first traffic recording must have been fluky. After creating a more isolated setup, it turns out my browser does recycle the TLS connection for both hostnames and does not even send a second SNI.

So in general, I am the idiot here, sorry for raising a false alarm. :)

Regarding my suggestion for the docs: Yes, the docs are correct in everything they say. But I believe this is a specific that might deserve explicit mention, I would not expect everyone who has to deal with SNI to read the entire wiki article / spec.

As a suggestion:

$ssl_server_name
    returns the server name requested through SNI (1.7.0); Note that this is not 
    necessarily the same as the http host in the requested URL.  

Again: I made a misguided assumption, but I think it was easy enough to fall for that trap. I believe that helping a future person avoid that trap is worth a commit. I'd do it myself, but I haven't found a way to do that.

Thanks for hearing me out!