mTLS client verification fails in >=1.21.4

[boris_sandler.cysiv.com@…]

I am using Nginx for performing mTLS authentication, terminating it and forwarding incoming connections to the backend.

I have been using Nginx docker images tagged with 'stable' until around Dec 21,2021 when the 1.21.4 image was tagged as 'stable', at which point all mTLS connections started failing in production. We had to revert to 1.21.3 to make them work again.

Nginx config looks like this:
worker_processes 1;

error_log /dev/stdout info;
pid /tmp/nginx.pid;
events {

worker_connections 1024;

}
stream {

server {

listen 443 ssl;
proxy_pass some-backend-host:443;
ssl_certificate /etc/nginx/ssl/tls.crt;
ssl_certificate_key /etc/nginx/ssl/tls.key;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_session_cache shared:SSL:2m;
ssl_client_certificate /etc/nginx/ssl/ca_cert.pem;
ssl_verify_client on;
ssl_session_timeout 4h;
ssl_handshake_timeout 30s;

}

}

This is the error I'm seeing in nginx logs in versions >=1.21.4:
2022/03/16 16:02:11 [info] 22#22: *18 client 10.126.0.10:20695 connected to 0.0.0.0:443
2022/03/16 16:02:11 [info] 22#22: *18 client SSL certificate verify error: (18:self signed certificate) while SSL handshaking, client: 10.126.0.10, server: 0.0.0.0:443
2022/03/16 16:02:11 [crit] 22#22: *18 SSL_shutdown() failed (SSL: error:14094123:SSL routines:ssl3_read_bytes:application data after close notify) while SSL handshaking, client: 10.126.0.10, server: 0.0.0.0:443

ca_cert.pem is a self-signed root CA (x509 RSA).
The client authenticates using a key and a cert signed by the key & CA.
The problem is that the signed cert's Issuer is identical to Subject implying it's self-signed, which works well in 1.21.3, but is rejected in 1.21.4 and more recent versions as a self-signed cert.

It is signed as can be validated via the root CA, it is issuer=subject in the client cert that apparently causes the problem.

This is a breaking change that seems to have been inadvertently introduced in version 1.21.4.

Setting ssl_verify_client to optional_no_ca works around the error but then it is not mTLS anymore as the client may send no certs.

I couldn't find anything in the changelog to explain this. It does not seem to be related to a drop in support for NPN. I was able to recreate this with a simple python socket as a client. It also fails on python 3.10, where NPN was replaced with ALPN.

[Maxim Dounin]

Not sure I understand your certificate configuration. From the description it looks like you have certificates like the following:

• Client: subject=foo, issuer=foo
• Root CA: subject=foo, issuer=foo

That is, both client and root CA certificates use identical subjects, but at the same time these are distinct certificates, with distinct keys, correct? I don't think this will work with any nginx or any OpenSSL though.

Or you mean client uses the exact self-signed certificate which is in ca_cert.pem file? This should work fine with all versions though.

Could you please provide output of openssl x509 -text -noout -in ca_cert.pem and the same for the client certificate to better understand the configuration?

In either case, I don't think there are any relevant changes in nginx between 1.21.3 and 1.21.4. It is possible that something relevant was changed in OpenSSL though. Assuming you refer to the official nginx docker image as available on Docker Hub, the OpenSSL version was changed from OpenSSL 1.1.1d in nginx:1.21.3 to OpenSSL 1.1.1k in nginx:1.21.4:

$ docker run -it nginx:1.21.3 nginx -V | grep OpenSSL
built with OpenSSL 1.1.1d  10 Sep 2019
$ docker run -it nginx:1.21.4 nginx -V | grep OpenSSL
built with OpenSSL 1.1.1k  25 Mar 2021

It is known that certificate verification code was changed at least in OpenSSL 1.1.1i (see ticket #2319), and your configuration might be affected either by this change or some other changes in OpenSSL.

[boris_sandler.cysiv.com@…]

yes, that is correct. Client cert was signed by (self-signed) Root CA and has its own key.
The issuer and subject of the two certs are identical.

Here are the outputs of both certs:
ca_cert.pem:

Certificate:

Data:

Version: 3 (0x2)
Serial Number:

5f:59:c7:8e:2c:bd:c6:64:4b:e8:f5:3a:ca:a3:69:e1:d0:a5:35:20

Signature Algorithm: sha256WithRSAEncryption

Issuer: C=CA, ST=Ontario, L=Toronto, O=Org, CN=cbedfa2f65ad487995efd7597b51eb1c
Validity

Not Before: Feb 15 15:25:01 2022 GMT
Not After : Feb 13 15:25:01 2032 GMT

Subject: C=CA, ST=Ontario, L=Toronto, O=Org, CN=cbedfa2f65ad487995efd7597b51eb1c
Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)
Modulus:

00:e9:10:4d:20:f0:0d:14:0d:9f:ec:f2:28:e5:40:
f9:4d:bb:40:55:51:95:ea:f1:61:95:52:e4:99:1c:
d5:ea:f4:b8:29:aa:9f:3e:79:de:cc:75:9a:d6:09:
c6:63:f0:23:f5:0e:cf:0b:a3:42:6d:d2:b1:35:a5:
66:a7:b5:bf:67:97:d9:20:3e:b1:c1:f6:51:be:3e:
3d:01:c4:d4:11:c5:dd:2f:82:0a:db:e5:5f:27:b7:
2c:6a:9c:61:96:31:e5:a0:36:ac:e7:1e:6e:07:df:
c0:ca:b3:5a:d7:04:13:c5:8b:51:af:74:48:13:c2:
be:ad:27:e3:75:8a:4f:0a:0e:d6:21:83:33:0d:87:
7b:66:32:e1:4e:de:4c:4f:85:24:4a:e5:c8:d0:20:
9b:48:9c:fd:2d:e8:62:9d:fe:08:b2:28:26:d1:cb:
2f:98:9e:50:15:cd:d5:8b:5c:e0:12:d1:84:e3:52:
ea:9a:ca:49:96:cc:b4:83:6c:8b:47:29:ff:92:dd:
2e:e3:d9:a0:5c:10:aa:00:c5:f8:c2:b5:4d:20:97:
d9:a5:df:a7:f0:3c:7a:dd:3e:95:7b:55:8a:78:df:
78:46:88:d7:c8:67:a2:68:0c:d3:c4:6a:c9:64:82:
3b:6e:79:7d:d8:cc:b8:98:d9:b6:cd:fe:08:f4:81:
ac:df

Exponent: 65537 (0x10001)

X509v3 extensions:

X509v3 Basic Constraints: critical

CA:TRUE

Signature Algorithm: sha256WithRSAEncryption

a6:7c:4d:53:aa:82:65:71:38:92:30:46:f4:8e:f2:fc:0a:7e:
70:4a:a4:f8:47:cf:c7:bd:e7:22:be:42:66:32:a2:6e:13:74:
55:c0:3c:ec:47:be:3c:5a:aa:b9:2f:6a:e6:51:6e:85:58:c1:
f8:50:a4:a2:7b:8b:9d:6c:02:27:45:53:69:fd:1a:4c:d6:4f:
58:62:e3:95:b0:41:5d:81:87:14:7e:b4:dc:54:45:ad:ce:a0:
3e:f3:99:e2:6a:bc:a3:c5:d1:f1:63:d5:1d:b9:0f:57:47:83:
b7:df:ac:5c:fc:21:25:7e:d5:28:7c:6d:7a:1e:a2:bd:9e:63:
a5:ec:e7:41:0d:24:49:14:54:c5:ad:ad:70:c7:3d:62:22:99:
02:5b:b7:b6:f5:b9:dd:9f:f3:6c:be:ce:66:40:18:ae:a9:ea:
bb:ed:48:20:70:64:02:01:9e:15:a3:21:2b:32:ed:94:bb:da:
35:e1:31:b2:b3:20:d1:db:84:8a:98:4c:57:eb:a5:56:3a:e7:
63:fa:0c:eb:1c:90:84:b7:76:ed:bd:69:25:0f:e9:43:bf:d8:
9d:5f:e7:46:24:c0:b2:b1:ad:2f:80:79:49:9a:da:c9:c1:be:
71:92:15:83:0c:7a:6f:9f:c9:54:47:d4:d8:9f:66:b6:82:69:
7d:48:84:21

client_cert.pem:
Certificate:

Data:

Version: 3 (0x2)
Serial Number:

6d:b2:0a:6d:53:0f:87:0a:88:6a:a0:d4:f8:a1:a1:ce:a4:23:71:08

Signature Algorithm: sha256WithRSAEncryption

Issuer: C=CA, ST=Ontario, L=Toronto, O=Org, CN=cbedfa2f65ad487995efd7597b51eb1c
Validity

Not Before: Mar 18 15:14:16 2022 GMT
Not After : Mar 17 15:14:16 2025 GMT

Subject: C=CA, ST=Ontario, L=Toronto, O=Org, CN=cbedfa2f65ad487995efd7597b51eb1c
Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (2048 bit)
Modulus:

00:ad:31:2c:b4:72:e1:2a:35:93:37:42:c5:8d:3d:
8c:c1:4c:39:5c:cf:e2:76:b9:a3:37:5a:bf:ed:80:
7c:11:54:ae:f3:71:69:68:dd:27:cc:5b:1e:80:6e:
49:84:51:76:69:25:1c:ab:35:71:0c:d1:86:12:d1:
8e:71:59:4e:92:68:8e:30:4e:a6:20:5f:a7:03:97:
01:45:5f:ee:10:17:b9:6c:29:f5:fd:b3:b0:e7:ca:
a3:30:29:b9:96:81:9c:95:67:42:da:6d:be:29:b0:
3b:0f:af:67:30:fd:74:3a:43:7e:e1:8b:65:6a:32:
34:a4:f2:8e:56:b9:45:94:0d:4d:60:b5:71:0a:cd:
df:85:cf:69:3f:d2:b9:7c:23:48:7e:bf:40:f4:94:
91:ec:a9:6a:e2:68:75:e2:ba:1e:7c:83:de:16:ba:
3c:20:6e:25:a4:bf:83:a0:e1:b8:84:c9:18:5f:ba:
c4:46:17:f6:e5:ac:f5:35:9f:f4:83:11:3c:fc:58:
7e:fa:b5:50:df:f9:50:d1:71:33:56:b5:e6:72:32:
02:20:05:34:ea:e0:40:51:98:d3:99:2a:22:83:0b:
83:a9:9d:fb:0b:85:83:49:f2:6c:d6:b0:e5:09:7f:
7b:23:4e:8c:d3:1f:e0:06:ea:71:f5:2a:0c:ed:54:
30:eb

Exponent: 65537 (0x10001)

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment

X509v3 Extended Key Usage:

TLS Web Client Authentication

X509v3 Basic Constraints: critical

CA:FALSE

Signature Algorithm: sha256WithRSAEncryption

18:32:f2:b8:35:69:c7:70:d5:c8:1f:c8:47:34:b6:b0:a8:7d:
4d:8e:d4:52:98:65:b9:4f:62:29:e6:eb:e4:38:9e:4b:64:11:
8a:33:21:1b:b5:14:ee:89:41:44:ea:9c:4f:38:24:0c:ea:32:
b8:e1:9f:28:3d:52:2f:de:74:2b:e4:68:fa:5a:f3:9e:d8:e7:
ef:6e:5a:09:75:97:6c:20:0b:17:a7:35:2a:e8:cd:29:d1:6f:
b0:01:4d:9a:99:fd:b3:96:19:81:3f:9f:88:59:b9:b9:7c:09:
3f:e6:21:65:01:f2:fd:fa:9e:77:81:33:96:e8:2a:0d:45:32:
18:a8:6c:5b:01:b6:9c:e6:98:91:6f:ef:9a:61:39:71:f6:4d:
fe:2f:39:62:00:5a:bc:6c:df:1f:ef:39:50:62:5d:b0:a5:99:
e2:3b:fa:78:b5:ab:bc:67:6a:83:73:fc:b2:1f:30:f0:c8:59:
af:db:a2:40:78:05:2d:bc:c7:58:06:97:35:f8:5a:0b:3f:e8:
fd:60:02:d4:84:f7:c3:a8:ff:2d:2d:6d:b4:4c:32:63:62:1c:
25:d8:e5:5d:9f:6f:1f:04:97:77:cf:cb:0f:87:9c:7b:c5:15:
f4:ba:92:a8:e9:c2:4c:01:a7:e1:83:64:74:8e:51:14:0d:e6:
4e:a4:86:7f

This works in 1.21.3, not in 1.21.4, probably due to the OpenSSL library upgrade as you mentioned.

If client_cert.pem is issued with any other subject, the authentication works in both 1.21.3 and 1.21.4.

[Maxim Dounin]

Thanks for the details.

With the provided key usage extensions I was able to reproduce it. Indeed, it works with OpenSSL 1.1.1g:

$ openssl-1.1.1g/.openssl/bin/openssl version
OpenSSL 1.1.1g  21 Apr 2020
$ openssl-1.1.1g/.openssl/bin/openssl verify -CAfile root.crt client.crt
client.crt: OK

But not with OpenSSL 1.1.1h:

$ openssl-1.1.1h/.openssl/bin/openssl version
OpenSSL 1.1.1h  22 Sep 2020
$ openssl-1.1.1h/.openssl/bin/openssl verify -CAfile root.crt client.crt
CN = foo
error 18 at 0 depth lookup: self signed certificate
error client.crt: verification failed

Given that this certificate configuration looks very strange, I'm not really sure it is supposed to work. On the other hand, it's probably up to OpenSSL developers to decide. In either case, this is certainly not an nginx issue, so closing this.