OCSP stapling non-functional on IPv6-only host

[ramcq@…]

I have an IPv6-only host running CentOS 7. I have a Lets Encrypt certificate on the host and I've enabled OCSP stapling per the Mozilla preferred SSL stuff. My provider has NAT64 set-up so I've configured their NAT64 resolvers in the resolve entry in nginx.conf.

        # OCSP Stapling ---
        # fetch OCSP records from URL in ssl_certificate and cache them
        ssl_stapling on;
        ssl_stapling_verify on;

        # verify chain of trust of OCSP response using Root CA and Intermediate certs
        ssl_trusted_certificate /etc/dehydrated/certs/flathub.org/chain.pem;

        resolver [2a00:1098:0:80:1000:3b:0:1] [2a00:1098:0:82:1000:3b:0:1];

I see this error:

2017/07/24 14:02:23 [error] 16637#0: connect() to 88.221.134.147:80 failed (101: Network is unreachable) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org

I believe that it's because this host returns two A and two AAAA results:

[root@front nginx]# host ocsp.int-x3.letsencrypt.org
ocsp.int-x3.letsencrypt.org is an alias for ocsp.int-x3.letsencrypt.org.edgesuite.net.
ocsp.int-x3.letsencrypt.org.edgesuite.net is an alias for a771.dscq.akamai.net.
a771.dscq.akamai.net has address 88.221.134.114
a771.dscq.akamai.net has address 88.221.134.147
a771.dscq.akamai.net has IPv6 address 2a02:26f0:e8::6856:6fb0
a771.dscq.akamai.net has IPv6 address 2a02:26f0:e8::6856:6f88

However the SSL stapling code only attempts to connect the first one: ​https://github.com/nginx/nginx/blob/9197a3c8741a8832e6f6ed24a72dc5b078d840fd/src/event/ngx_event_openssl_stapling.c#L1028

I've tried to work around with /etc/hosts but that seems unused, and OCSP stapling seems to disable itself if I have no resolver configuration entry. I can't seem to place an IPv6 address in the ssl_stapling_responder either.

[Maxim Dounin]

The problem is as follows:

• OCSP stapling connection handling is very basic, and simply uses the first address returned. This is enough in most cases, even if some of the addresses returned are not reachable, because addresses returned by a resolver usually rotated.
• This does not work with IPv6-only hosts though, as for compatibility reasons nginx always places IPv4 addresses first.

As a result, as long as a name resolves to both IPv4 and IPv6 addresses, an IPv4 address is always tried first. It is not reachable though, hence OCSP request fails.

The following workarounds should work out of the box though:

• Remove the resolver directive from the configuration, and configure system resolver to only IPv6 addresses (e.g., via /etc/hosts). This way nginx will complain about ... [warn] ... no resolver defined to resolve ... on each request to the OCSP responder, though should work fine using addresses obtained from system resolver during configuration parsing.
• Configure ssl_stapling_responder with an reachable address. Note though, that just using an IPv6 address of a real responder likely won't work, as the resulting request will contain an invalid Host header and likely will be rejected. It can be easily worked around using nginx itself though, with something like this:

  server {
      ...
      ssl_stapling on;
      ssl_stapling_responder http://[::1]:8082;
      ...
  }
  
  server {
      listen [::1]:8082;
  
      location / {
          proxy_pass http://ocsp.int-x3.letsencrypt.org;
      }
  }
  

  Proxy is smart enough to re-try requests if a connection fails, so it should work without additional filtering of addresses.

Proper fix would be to introduce fallback to different addresses in OCSP and/or something like resolver ... ipv4=off (similar to resolver ... ipv6=off we have now).

[Maxim Dounin]

See also #1363.

[v10lator.myway.de@…]

I wanted to add another workaround: Use a DNS A record filter proxy ( ​https://tools.ietf.org/html/draft-hazeyama-sunset4-dns-a-filter-00 ).
Sadly I couldn't find any implementation for this, so I used the howto at ​https://peteris.rocks/blog/dns-proxy-server-in-node-js-with-ui/ and changed the codes a bit. Here's the result: ​https://pastebin.com/W49EGiMV

So far this seems to have no drawbacks when used on a IPv6 only server but fixes quirks in other software, too.

EDIT: New codes: ​https://pastebin.com/2NTfJu84

[ramcq@…]

Replying to mdounin:

The following workarounds should work out of the box though:

• Configure ssl_stapling_responder with an reachable address. Note though, that just using an IPv6 address of a real responder likely won't work, as the resulting request will contain an invalid Host header and likely will be rejected. It can be easily worked around using nginx itself though, with something like this:

  server {
      ...
      ssl_stapling on;
      ssl_stapling_responder http://[::1]:8082;
      ...
  }
  
  server {
      listen [::1]:8082;
  
      location / {
          proxy_pass http://ocsp.int-x3.letsencrypt.org;
      }
  }
  

This workaround worked, with one tweak - as I mentioned in the original ticket the ssl_stapling_responder seemed unwilling to accept a bare IPv6 address in []s, so I put ​http://localhost in there instead.

[Maxim Dounin]

Replying to ramcq@…:

This workaround worked, with one tweak - as I mentioned in the original ticket the ssl_stapling_responder seemed unwilling to accept a bare IPv6 address in []s, so I put ​http://localhost in there instead.

You have to use URL, not just an IP address. The configuration as written (with ssl_stapling_responder http://[::1]:8082;) works fine here.

[Roman Arutyunyan]

In 7652:7cffd81015e7/nginx:

OCSP stapling: iterate over all responder addresses.

Previously only the first responder address was used per each stapling update.
Now, in case of a network or parsing error, next address is used.

This also fixes the issue with unsupported responder address families
(ticket #1330).

[Maxim Dounin]

For the record.

It looks like my initial analysis in comment:1 was wrong, and OCSP stapling should be working fine as long as the resolver directive was used in the configuration, since resolver actually rotates all IP addresses, not preferring neither IPv4 nor IPv6. While the error itself is expected to appear, subsequent OCSP stapling requests are likely to use other addresses and succeed.

The error might be fatal when not using the resolver directive though, since only the first IP address was used then (obtained during configuration parsing). Before nginx 1.15.10 (8be88b22fe81) it resulted in stapling not working on IPv6-only hosts unless the system resolver was configured to return only IPv6 addresses. Starting with 1.15.10 it might result in fatal errors if IPv4 address was returned first by the system resolver on an IPv6-only host, or IPv6 address was returned first on an IPv4-only host.

With the patch committed all IP addresses will be tried in any case, eliminating the problem. Note though that errors might still appear in logs as long as some addresses tried are not reachable.