Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#1449 closed defect (worksforme)

ocsp failed, nginx failed to establish new connections

Reported by: https://stackoverflow.com/users/1100117/higuita Owned by:
Priority: major Milestone:
Component: other Version: 1.10.x
Keywords: Cc:
uname -a: Linux nginxlb--i-0ffcc4148076db4c9 4.9.0-4-amd64 #1 SMP Debian 4.9.51-1 (2017-09-28) x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.10.3
built with OpenSSL 1.1.0f 25 May 2017
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-2tpxfc/nginx-1.10.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module --add-dynamic-module=/build/nginx-2tpxfc/nginx-1.10.3/debian/modules/nginx-auth-pam --add-dynamic-module=/build/nginx-2tpxfc/nginx-1.10.3/debian/modules/nginx-dav-ext-module --add-dynamic-module=/build/nginx-2tpxfc/nginx-1.10.3/debian/modules/nginx-echo --add-dynamic-module=/build/nginx-2tpxfc/nginx-1.10.3/debian/modules/nginx-upstream-fair --add-dynamic-module=/build/nginx-2tpxfc/nginx-1.10.3/debian/modules/ngx_http_substitutions_filter_module

Description

Using the mozilla config generator, https://mozilla.github.io/server-side-tls/ssl-config-generator/ , i have the ssl stampling.
Today the server stop receiving connections and in the logs i got this:

2017/12/16 13:31:33 [error] 2069#2069: ocsp.comodoca.com could not be resolved (110: Operation timed out) while requesting certificate status, responder: ocsp.comodoca.com
2017/12/16 13:36:16 [error] 2069#2069: unexpected response for ocsp.comodoca.com
2017/12/16 13:36:16 [error] 2069#2069: unexpected response for ocsp.comodoca.com
2017/12/16 13:36:16 [error] 2069#2069: unexpected response for ocsp.comodoca.com
2017/12/16 13:36:48 [error] 2069#2069: ocsp.comodoca.com could not be resolved (110: Operation timed out) while requesting certificate status, responder: ocsp.comodoca.com
2017/12/16 13:36:57 [error] 2069#2069: unexpected response for ocsp.comodoca.com
2017/12/16 13:36:58 [error] 2069#2069: unexpected response for ocsp.comodoca.com
2017/12/16 13:36:58 [error] 2069#2069: unexpected response for ocsp.comodoca.com
2017/12/16 13:43:03 [error] 2069#2069: ocsp.comodoca.com could not be resolved (110: Operation timed out) while requesting certificate status, responder: ocsp.comodoca.com
2017/12/16 13:55:32 [error] 2069#2069: ocsp.comodoca.com could not be resolved (110: Operation timed out) while requesting certificate status, responder: ocsp.comodoca.com
2017/12/16 13:55:43 [error] 2069#2069: unexpected response for ocsp.comodoca.com
2017/12/16 13:55:43 [error] 2069#2069: unexpected response for ocsp.comodoca.com

restarting the nginx was enough to solve this... but of course, nginx should not lock up with the ocsp fails

Change History (2)

comment:1 by Maxim Dounin, 3 years ago

Resolution: worksforme
Status: newclosed

Messages suggests that nginx was not able to resole the OCSP responder name - from the logs it looks like DNS responses were coming after nginx given up waiting for them. These and other OCSP-related errors will not prevent nginx from working though, it will continue handling connections without OCSP stapling.

The root cause of the name resolution problems though - either non-working DNS server as seen from logs, or may be some network problems which caused DNS to be unresponsive - might be the real reason of the observed connectivity issues. You may want to dig further to understand what actually happened with your server.

comment:2 by https://stackoverflow.com/users/1100117/higuita, 3 years ago

Thanks for the feedback!

Notice that i could connect to other services at that time on the same server and i already saw that error in the past without any nginx problem (so proving that this alone should not be a problem), so i suspected a non-usual path/response that locked the nginx ssl part.

i was able to see that nginx got stuck "slowly", it could still response, but after some time no more new connections were accepted. I would say that keepalive connections where working, but something was locking the creating of new ssl connections.

we have another nginx in the same network and that one was working fine during the same time.

If it happens again, i will try to do a strace and a gdb to the nginx to try to give more hints what is happening

Note: See TracTickets for help on using tickets.