OCSP verification fails if response is signed by a designated authority

[Yan Foto]

Enabling stapled OCSP by setting ssl_stapling and ssl_stapling_verify works flawlessly if the certificate and its OCSP response are both signed by the same authority, i.e. identified by the same certificate. This is how Let's Encrypt issues and signs OCSP certs, e.g. both by issuer=C = US, O = Let's Encrypt, CN = R3.

If, however, the OCSP responce is signed by a designated authority which is not the same as the issuing CA (and is not included in chain passed to ssl_certificate), nginx fails to verify the OCSP response:

OCSP_basic_verify() failed (SSL: error:27069065:OCSP routines:OCSP_basic_verify:certificate verify error:Verify error:unable to get local issuer certificate) while requesting certificate status, responder: ocsp.buypass.com, peer: 23.55.161.81:80, certificate: "/etc/letsencrypt/live/example.com/fullchain.pem"

Nonetheless, RFC 2560 (see ​https://tools.ietf.org/html/rfc2560#section-4.2.2.2) considers such situation as valid and thus should be properly handled by nginx.

---

I have tested this with DV certificates generated and got signed using certbot, one from Let's Encrypt (works) and another from buyssl (fails). Certbot generates three files:

• chain.pem: intermediate CA certs
• fullchain.pem: leaf + intermediate CA certs
• privkey.pem: private key

For Let's Encrypt certs I only need the following lines in my config:

    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; # managed by Certbot

For buyssl certs I need to add an extra line:

ssl_trusted_certificate /etc/letsencrypt/live/example.com/ocsp-chain.pem;

where ocsp-chain.pem is fullchain.pem + the root certificate of buyssl which is already included in my /etc/ssl/certs. In other words I have to pass the whole trust chain for nginx to be able to verify OCSP.

[Yan Foto]

I also get the same results with nginx/1.18.0:

nginx version: nginx/1.18.0
built by gcc 8.3.0 (Debian 8.3.0-6) 
built with OpenSSL 1.1.1d  10 Sep 2019
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-1.18.0/debian/debuild-base/nginx-1.18.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

[Maxim Dounin]

Unfortunately, OpenSSL provides no easy way to verify OCSP responses despite the fact that all valid responses are expected to be verifiable using only the issuer certificate. Instead, it requires full certificate chain up to the root certificate to verify such responses, so you have to provide all relevant certificates (notably the root certificate, as others are usually available in the certificate chain and nginx is smart enough to use them) using the ssl_trusted_certificate directive. This is explicitly documented in the ​ssl_stapling_verify directive description:

For verification to work, the certificate of the server certificate issuer, the root certificate, and all intermediate certificates should be configured as trusted using the ssl_trusted_certificate directive.

And this is the main reason why ssl_stapling_verify is off by default.

[Yan Foto]

I appreciate the quick response, but I'm not sure if that's an OpenSSL issue. If you use the CLI as follows:

openssl ocsp -issuer chain.pem -cert fullchain.pem -url "${OCSP_URL}"

it will properly validate the response in either case, i.e., the response is signed by the issuing CA or a designated authority. In addition, in the solution that I mentioned above I don't mark the certificate of the designated authority as trusted but its parent (root anchor). At the same time (despite what the documentation says) nginx is well able to verify Let's Encrypt OCSP resoponces without explicitly including its respective root.

I assume if nginx follows the ​steps as the OpenSSL CLI there would be no need to explicitly include an anchor which is already part of the system wide trust store.

[Yan Foto]

Another note:

[...] despite the fact that all valid responses are expected to be verifiable using only the issuer certificate.

Is not exactly what ​RFC 2560 states:

The key that signs a certificate's status information need not be the same key that signed the certificate. A certificate's issuer explicitly delegates OCSP signing authority by issuing a certificate containing a unique value for extendedKeyUsage in the OCSP signer's certificate. This certificate MUST be issued directly to the responder by the cognizant CA.

If you look at how OpenSSL CLI ​verifies the OCSP response, you can see that it only passes the X509 stack of issuer(s) (what you pass as -issuer) and the root store to OCSP_basic_verify(). In fact, if we get back to my example, passing the cert and its issuer is enough for the OpenSSL OCSP CLI to verify:

openssl ocsp -issuer chain.pem -cert cert.pem -url "${OCSP_URL}"

In addition, as the ​OpenSSL `OCSP_basic_verify` doc states, there is no need to explicitly pass the certificate of designated authority, i.e., response signer, if it differs from the issuer:

The function first tries to find the signer certificate of the response in <certs>. It also searches the certificates the responder may have included in bs unless the flags contain OCSP_NOINTERN.

I have been reading the OpenSSL and NGINX source code for the past hours and a proper debugging seems to be beyond my abilities, but I can imagine that ​these lines should be adapted to follow the logic ​here:

i = OCSP_basic_verify(bs, verify_other, store, verify_flags);
if (i <= 0 && issuers) {
    i = OCSP_basic_verify(bs, issuers, store, OCSP_TRUSTOTHER);
    if (i > 0)
        ERR_clear_error();
}
if (i <= 0) {
    BIO_printf(bio_err, "Response Verify Failure\n");
    ERR_print_errors(bio_err);
    ret = 1;
} else {
    BIO_printf(bio_err, "Response verify OK\n");
}

That is first to check against any explicitly indicated validator certificate files and if it fails, verify against the issuer certificates.

[Yan Foto]

For future reference:

I spent some more time trying to figure out what is different in NGINX and observed a very strange behavior. Although, OCSP_BASICRESP *bs (response) and STACK_OF(X509) (issuer(s)) parameters passed to OCSP_basic_verify are the same as what the CLI uses, if you print out the response (I used OCSP_RESPONSE_print to append it to logs), it does not print the certificate that is returned from the server (the CLI does), as if the response doesn't carry any extra certs (see ​here for implementation).

This can explain why NGINX fails to verify buypass responses but succeeds in verifying those by Let's Encrypt: buypass uses a designated authority to sign OCSP responses which differs from the certificate issuer (see my previous comments).

[Maxim Dounin]

there would be no need to explicitly include an anchor which is already part of the system wide trust store

The key part here is "system wide trust store" you are referring to. As nginx does not use "system wide trust store", you have to configure trusted certificates using the ssl_trusted_certificates directive.

Is not exactly what ​RFC 2560 states

Note that it "MUST be issued directly to the responder by the cognizant CA". Or, more strictly in ​RFC 6960, "MUST be issued directly by the CA that is identified in the request". This implies that an OCSP response can be verified by using only issuer certificate ("the CA that is identified in the request") and, if designated responder is used, by the certificate of the designated responder, which is expected to be included in the response.

[Yan Foto]

This implies that an OCSP response can be verified by using only issuer certificate

Yes. After posting my previous comment, I also realized that we are talking about the same thing.

---

I have now found out what the problem is! As I mentioned in my previous comment, the server returns an additional certificate which seems to be ignored by OpenSSL contrary to what its documentation says (see my previous comment). So If you manually pass this extra cert to OCSP_basic_verify alongside the issuer certificate, lo and behold, the verification succeeds and NGINX manages to staple the response correctly.

I have generated ​a patch for `src/event/ngx_event_openssl_stapling.c` which although not suitable for production can show you how my solution works*. In sum, it all comes down to the following:

STACK_OF(X509) *issuers = NULL;
issuers = sk_X509_new_null();
sk_X509_push(issuers, ctx->issuer);
for (int i = 0; i < sk_X509_num((STACK_OF(X509) *) basic->certs); i++) {
    sk_X509_push(issuers, sk_X509_value((STACK_OF(X509) *) basic->certs, i));
}

OCSP_basic_verify(basic, issuers, store, ctx->flags);

This is exactly what I expect OpenSSL to do automatically when verifying the basic reponse (as its CLI does).

Finally, I assume that this is a bug with OpenSSL and not NGINX (so technically not belonging here), but at least I know for sure that was not wrong in assuming that I don't need to pass any extra certificates (thru ssl_trusted_certificate) for the OCSP verification to work.

Update:

I just wrote a small program to check OpenSSL API. I could verify that in my small example OpenSSL could easily read and print the certificate included in the OCSP response. So I guess the question remains: why doesn't it manage to do the same in NGINX context?

---

. * I use libssl-dev 1.1.1d where you need to symlink/copy crypto/ocsp/ocsp_lcl.h for my patch to work as the struct definitions here seem to be private.

[Maxim Dounin]

So I guess the question remains: why doesn't it manage to do the same in NGINX context?

As far as I understand your code, it is essentially equivalent to no verify at all, since signing an OCSP response with any certificate sent along with it will be enough for verification to succeed. This is what nginx does by default, with ​ssl_stapling_verify off;. That is, no additional code is needed, just revert the incorrect ssl_stapling_verify configuration you've added to your config.

In general it is certainly possible to do things properly by re-implementing OCSP verification in nginx. It is, however, believed to be something for OpenSSL to fix.

[Yan Foto]

This is summary of of the discussion so far and my final comment.

tl;dr

OpenSSL is not used/configured properly in NGINX as OCSP_basic_verify() does not regard certificates included in the OCSP response (use OCSP_RESPONSE_print to verify).

How to reproduce:

1. Choose a provider that uses different entities (i.e., identified by different certs) to issue certificates and verify respective OCSP responses (I used ​buypass). Retrieve a (test/staging) certificate and verify that OpenSSL can validate OCSP response:

openssl ocsp -issuer issuer.pem -cert cert.pem -url "${OCSP_URL}" -text

Note the -text argument that directs OpenSSL to print OCSP request, response, and any certificates delivered by the OCSP responder and verify that buypass actually delivers an extra certificate signed by the certificate issuer to verify the OCSP response.

2. Configure your NGINX server to enable OCSP stapling and verification without explicitly including the respective root certificate using ssl_trusted_certificate. If you are not sure if this configuration is valid at all, you can test it with a certificate issued by Let's Encrypt where issuer CA is the same as OCSP responder (i.e., no extra certs in OCSP response). Verify that NGINX fails to validate OCSP response.

3. Just ​before `OCSP_basic_verify()` is called add OCSP_RESPONSE_print and verify that this time OpenSSL does not print the additional cert included in the response. If you want to make sure that the OCSP responder has included the certificate, you can use wireshark to sniff the response. You can also count how many certs are inside basic -> certs (exactly one in this case).

4. Try to imitate ​what OpenSSL internally does in `OCSP_basic_verify`, i.e., taking the extra cert into consideration, before NGINX calls the OCSP_basic_verify and pass this new stack:

STACK_OF(X509) *issuers = sk_X509_dup(ctx->chain);
for (int i = 0; i < sk_X509_num((STACK_OF(X509) *) basic->certs); i++) {
    sk_X509_push(issuers, sk_X509_value((STACK_OF(X509) *) basic->certs, i));
}
OCSP_basic_verify(basic, issuers, store, ctx->flags);

Verify that NGINX now succeeds to validate the response. NOTE that this code is only to be understood as a proof to show that OpenSSL is not working properly in the context of NGINX. In a proper setting OpenSSL does this correctly with some criticial checks that I omitted here for the sake of presentation.

Some notes:

As far as I understand your code, it is essentially equivalent to no verify at all

No it's not. This is a poor man's version of what ​OpenSSL internally does, ignoring some extra steps and checks:

1. Read certificate included in the OCSP response and consider them as untrusted.
2. Check if this cert conforms to RFC specifications, i.e. OCSP signer flag is set and its signed by the issuer. Consequently construct a chain of [issuer, OCSP responder].
3. Use this chain to verify the OCSP response.

Just revert the incorrect ssl_stapling_verify configuration you've added to your config.

That would be erasing the question altogether.

In general it is certainly possible to do things properly by re-implementing OCSP verification in nginx. It is, however, believed to be something for OpenSSL to fix.

I'm convinced that this is not an OpenSSL problem:

1. Write a small program using OpenSSL to verify OCSP responses (in DER format).
2. Dump the response retrieved by NGINX and feed it to your program to see that the verification works.
3. Conclude that OpenSSL works as it should!

---

It's futile to continue this thread. You can follow the steps I mentioned above and see for yourself that it's a valid issue. Everything else is speculation.

[Maxim Dounin]

Replying to Yan Foto:

1. Choose a provider that uses different entities (i.e., identified by different certs) to issue certificates and verify respective OCSP responses (I used ​buypass). Retrieve a (test/staging) certificate and verify that OpenSSL can validate OCSP response:

openssl ocsp -issuer issuer.pem -cert cert.pem -url "${OCSP_URL}" -text

Note the -text argument that directs OpenSSL to print OCSP request, response, and any certificates delivered by the OCSP responder and verify that buypass actually delivers an extra certificate signed by the certificate issuer to verify the OCSP response.

Note that this will fail to verify the OCSP response _unless_ you have appropriate root certificate in the standard OpenSSL certificate store. Try using -CAfile and -CApath arguments explicitly to see what happens if there is no root certificate in the store. This is exactly equivalent to nginx behaviour: it fails to verify an OCSP response unless you have appropriate root certificate in the certificate store set by ssl_trusted_certificate directive.

As far as I understand your code, it is essentially equivalent to no verify at all

No it's not. This is a poor man's version of what ​OpenSSL internally does, ignoring some extra steps and checks:

1. Read certificate included in the OCSP response and consider them as untrusted.
2. Check if this cert conforms to RFC specifications, i.e. OCSP signer flag is set and its signed by the issuer. Consequently construct a chain of [issuer, OCSP responder].
3. Use this chain to verify the OCSP response.

There is no step in your code which checks that the certs from the OCSP response are signed by the issuer. This means that any untrusted certificate can be used to sign the response. This is equivalent to not checking if the response is signed by a trusted certificate, much like what happens by default with ssl_stapling_verify off;.

To further clarify:

1. You put all certificates from the OCSP response into the issuers stack, which is then passed in the certs argument of OCSP_basic_verify().
2. The ret = ocsp_find_signer(&signer, bs, certs, flags); ​call in OCSP_basic_verify() will result in signer certificate being found in certs (instead of bs->certs), and hence ret set to 2.
3. As per ((ret == 2) && (flags & OCSP_TRUSTOTHER)) in the ​following lines, the OCSP_NOVERIFY flag will be set.

That is, the net effect of your code is that the OCSP_NOVERIFY flag is set when the OCSP_basic_verify() checks the flag. This is exactly equivalent to what ssl_stapling_verify off; does.

In general it is certainly possible to do things properly by re-implementing OCSP verification in nginx. It is, however, believed to be something for OpenSSL to fix.

I'm convinced that this is not an OpenSSL problem:

1. Write a small program using OpenSSL to verify OCSP responses (in DER format).
2. Dump the response retrieved by NGINX and feed it to your program to see that the verification works.
3. Conclude that OpenSSL works as it should!

Feel free to follow your own advice and provide a verification code which shows that "OpenSSL works as it should". Note that I already wrote the code to verify OCSP responses - the one in nginx - and it clearly shows that OpenSSL doesn't do what it should. And OpenSSL's code shows the same: OpenSSL cannot build a trust chain up to the issuer cert using certificates from the OCSP response. Instead, it provides only two options: it can explicitly trust the signers certificate if it is in the list (and this works well when the OCSP response is signed directly by the issuer), or it can verify full trust chain up to the root certificate.

It's futile to continue this thread. You can follow the steps I mentioned above and see for yourself that it's a valid issue. Everything else is speculation.

Sure. And the proper solution would be to fix OpenSSL's OCSP response verification to be able to verify a response using only issuer certificate. That's what essentially was said in the comment:1, along with two possible workarounds - use ssl_stapling_verify off; (which is the default) or provide certificates needed by OpenSSL via ssl_trusted_certificate (which is equivalent to what happens with openssl ocsp ... when you have appropriate root certificate in the standard OpenSSL certificate store). If you are not happy with the workarounds, consider working on appropriate OpenSSL improvements.

[Yan Foto]

Checkmate!

---

Try using -CAfile and -CApath arguments explicitly to see what happens if there is no root certificate in the store.

I did that already using -CAfile chain.pem, missing the fact that if -no-CApath is not explicitly set it will still include your system root store.

If you are not happy with the workarounds, consider working on appropriate OpenSSL improvements.

I am happy with NGINX, no question. And I should most probably starting fixing OpenSSL.

---

Sorry for wasting your time.