ssl_ecdh_curve is sometimes ignored in server blocks

[Avamander]

Consider a scenario when a single IP x.y.z.q has two server blocks. Both server blocks listen on the same port and support TLS. One of those blocks is marked default_server and handles the non-SNI requests.

If both blocks define ssl_ecdh_curve then it has zero effect on the non-default_server. This is done without warning.

One of the possible implications of this is that a more secure configuration is silently ignored. (I stumbled upon this when trying to enable post-quantum key exchange algorithms.)

Understandably nginx can't (currently, even though server_name could be read before KeXs come into play) respect the directive in both blocks, but in that case the ignored one should throw a non-critical warning. Plus, it could be better-documented.

[Maxim Dounin]

Aspects of how settings are applied to name-based virtual servers are documented in the ​Virtual server selection section of the ​Server names article. The general rule is that in name-based virtual servers custom settings might not be used if the particular operation which uses the setting happens before the name is known: in this case settings from the default server for the listening socket apply instead.

In case of SSL handshakes, this is further complicated by the fact that SSL handshakes are handled by the OpenSSL library, and OpenSSL might use settings from the default server even if not strictly required to happen before the name is known. For example, SSL protocol is selected by OpenSSL at a very early stage of the handshake, and it is not possible to change it afterwards, when the name is known (see #676 for details).

In the particular case of ​ssl_ecdh_curve, things looks as follows:

• In OpenSSL before 1.0.2 (before introduction of curve lists), the specified setting properly applied by OpenSSL to name-based virtual servers (as it is stored in cert->ecdh_tmp, and certificates are properly replaced during the SSL context switch in servername callback).

• In OpenSSL 1.0.2 and above, the list of configured curves is copied from the SSL context when the connection is created (see SSL_new()). As such, configuration from the default server applies to all name-based virtual servers. It is possible to re-apply the setting to the connection in the servername callback though (but see below).

• In OpenSSL 1.1.1 and above (tested up to OpenSSL 3.1.2) when using TLSv1.3, attempts to re-apply the setting results in handshake failure on the client:

  $ openssl s_client -connect 127.0.0.1:8443 -servername p384 -curves P-384
  CONNECTED(00000003)
  00000000:error:0A00006C:SSL routines:tls_parse_stoc_key_share:bad key share:ssl/statem/extensions_clnt.c:1771:
  

Relevant error details from the OpenSSL code:

        /*
         * It is an error if the HelloRetryRequest wants a key_share that we
         * already sent in the first ClientHello
         */
        if (group_id == s->s3.group_id) {
            SSLfatal(s, SSL_AD_ILLEGAL_PARAMETER, SSL_R_BAD_KEY_SHARE);
            return 0;
        }

That is, it looks like such a server emits HelloRetryRequest when it shouldn't.

• In BoringSSL, the list of configured curves is copied from the SSL context when the connection is created (similarly to OpenSSL 1.0.2, see SSL_new()). It does, however, work properly with TLSv1.3 when redefined in the servername callback.

• In LibreSSL (tested with LibreSSL 3.8.0), the list of configured curves is copied from the SSL context when the connection is created (similarly to OpenSSL 1.0.2, see SSL_new()). With TLSv1.2, it does work properly when redefined in the servername callback. With TLSv1.3, attempts to redefine the list of curves in the servername callback are ignored, the configuration from the original SSL context is used instead.

Overall, it might be possible to improve things by re-applying the list of curves in the servername callback. This might, however, cause degradation of some perfectly valid configurations with TLSv1.3 and OpenSSL. Also, it is not clear if it is actually needed: starting with introduction of curve lists in OpenSSL 1.0.2, I've hardly seen usage of non-default ssl_ecdh_curves which actually make sense in production setups.

[Avamander]

I guess the default auto is totally fine in most cases, though when one intentionally does specify a new configuration, it would be better to follow it.

In my case I specifically wanted to enable additional (quantum-resistant) curves for testing and it was really annoying to debug.

Though, this seems very fragile to truly fix and might cause more confusion. I suspect a simple warning when starting the server or in the documentation would suffice in practice.

[Maxim Dounin]

Note that nginx makes no real distinction between name-based virtual servers and IP/port-based virtual servers. Further, the same "server" block can be both name-based and IP-based (for different listening sockets) at the same time. Not to mention that actual behaviour depends on the undocumented aspects of handshake handling within the particular SSL library being used. As such, I'm highly sceptical about warnings. In general, this is not something nginx does for such settings.

General documentation is already present in the article mentioned. It probably can be improved though.