ssl_reject_handshake on introduced in nginx 1.19.4 works but disables TLSv1.3?

[i81b4u@…]

After "adding ssl_reject_handshake on;" to my configuration, everything seemed to work as intended, but after closer inspection I found that TLSv1.3 was no longer used.

After disabling the first server directive and restarting nginx, TLSv1.3 started working again:
server {

listen 443 ssl http2;
listen [::]:443 ssl http2;

ssl_reject_handshake on;

}

server {

listen 443 ssl http2;
listen [::]:443 ssl http2;

server_name test.server;

# logging
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log warn;

set $base /var/www/default;
root $base;

# SSL
ssl_certificate /opt/nginx/etc/ssl/certs/test.server.pem;
ssl_certificate_key /opt/nginx/etc/ssl/keys/test.server.key;

# index.html
index index.html;

# limit HTTP methods
location / {

limit_except GET POST {

deny all;

}

}

location ~* (mp3|mp4|m4v)$ {

add_header Content-Disposition "attachment";

}

}

The ssl-config is as follows:

ssl_session_timeout 1d;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
ssl_dyn_rec_enable on;
ssl_ecdh_curve X25519:secp521r1:secp384r1:prime256v1;
ssl_prefer_server_ciphers on;
ssl_buffer_size 4k;

ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE+AESGCM;

[Sergey Kandaurov]

Thank you for your report. This is a known issue.

OpenSSL checks server supported versions while processing
ClientHello and before its extensions such as server_name.
So that OpenSSL uses a default context for that check.
Then, OpenSSL removes TLSv1.3 version support since there's
no certificates loaded. For details, see its libssl function
is_tls13_capable().

This is believed to be a bug in OpenSSL, since it checks
for supported protocol versions before switching context,
and nginx can't influence it.
For comparison, BoringSSL works here as expected.

As a workaround, you may want to specify certificates
in the server block that has ssl_reject_handshake, too,
or switch to BoringSSL if that's an option.

[i81b4u@…]

Thanks a lot for your quick response. Much appreciated!

In the mean time I submitted a bug report with openssl and after recompiling nginx with boringssl support, ssl_reject_handshake works like a charm. Cool!

[Maxim Dounin]

For the record: ​https://github.com/openssl/openssl/issues/13291

[Maxim Dounin]

See also #2073.

[IlyaIndigo@…]

Then in the description of the option it is necessary to mention "Attention! For BoringSSL only!".

[i81b4u@…]

Hmmm, a member of the OpenSSL development team did implement a fix (master: ​https://github.com/openssl/openssl/pull/13304 and 1.1.1: ​https://github.com/openssl/openssl/pull/13305). Its state is "hold: need otc decision", but I can tell you that it fixes the problem, because I now have everything working with nginx and openssl 1.1.1h including those fixes.

The way I see it, the development team faces an almost philosophical question. Will they follow the pragmatic track and approve the fix "as is" (exception management), or will they rewrite part of the code in a way that no exception is needed.

If you ask me, I would implement the current fix and put the task of rewriting part of the code on the (already extensive) to-do list. But that's just my view on the matter and I'm not a member of the otc.

[i81b4u@…]

FYI: I saw today that the code that fixes the problem made it into both the master branch as well as the 1_1_1 branch of openssl.

[Maxim Dounin]

See also ticket #2101 comments.

[Maxim Dounin]

See also #2305.