Incorrect TLS session resumption for SNI-enabled upstream.

[oleg.guba@…]

I use configuration like this:

http {
    upstream upstream.domain.com {
        keepalive 128;
        server 1.1.1.1:443 max_fails=0;
    }
    server {
        listen 1.2.3.4:443 ssl;
        ssl_certificate x.domain.com.crt;
        ssl_certificate_key x.domain.com.key;

        server_name x.domain.com;
        location / {
            proxy_pass https://upstream.domain.com;
        }
    }
    server {
        listen 1.2.3.4:443 ssl;
        ssl_certificate y.domain.com.crt;
        ssl_certificate_key x.domain.com.key;
        server_name y.domain.com;
        location / {
            proxy_pass https://upstream.domain.com;
        }
    }
}

So nginx serves two different https servers with different certificates on one IP, both of them use same upstream.
Upstream 1.1.1.1:443 is also nginx-powered, has the same configuration with SNI (except the upstream host).

Using this configuration some (not all) requests to x.domain.com or y.domain.com fail with:

[error] 94934#0: *1388162830 upstream SSL certificate does not match "x.domain.com" while SSL handshaking to upstream, client: 10.48.0.154, server: x.domain.com, request: "POST /create HTTP/1.1", upstream: "https://1.1.1.1:443/create", host: "x.domain.com"

It fails here: ​https://github.com/nginx/nginx/blob/9edd64fcd842870ea004664288cadc344c33f0bd/src/event/ngx_event_openssl.c#L3177
I found that this error only happens when nginx tries to reuse ssl session to upstream.
I added some additional logging and found that sometimes reusing ssl session nginx gets incorrect cert (for upstream with server_name y.domain.com instead of x.domain.com).
Disabling ssl session reuse for upstream (proxy_ssl_session_reuse off) solves the problem.
I expect there is some issues with ssl session cache for SNI-enabled upstreams.

[Maxim Dounin]

SSL sessions are cached in the context of the upstream{} block (or an implicit upstream when using an IP address or a DNS name). That is, a configuration like this:

proxy_ssl_verify on;
proxy_ssl_server_name on;

location /x/ {
    proxy_pass https://x.example.com;
}

location /y/ {
    proxy_pass https://y.example.com;
}

is expected to work fine without any problems, and the error you see should never happen.

On the other hand, if you use the same upstream for multiple SNI names, the error is expected. That is, if you use something like:

proxy_ssl_verify on;
proxy_ssl_server_name on;

location /x/ {
    proxy_pass https://127.0.0.1;
    proxy_ssl_name x.example.com;
}

location /y/ {
    proxy_pass https://127.0.0.1;
    proxy_ssl_name y.example.com;
}

it will expectedly fail with session reuse, because sessions to https://127.0.0.1 will be cached regardless of SNI name and/or other SSL parameters used.

Given the above, I suspect that your proxying configuration is what causes the error you see. Please show your proxying configuration to confirm.

[Maxim Dounin]

Feedback timeout.