Opened 7 years ago
Closed 7 years ago
#1307 closed defect (invalid)
proxy_ssl_verify fails with multiple upstreams
| Reported by: | Owned by: | ||
|---|---|---|---|
| Priority: | blocker | Milestone: | |
| Component: | nginx-module | Version: | 1.11.x |
| Keywords: | ssl | Cc: | |
| uname -a: | Linux xxxxx 3.8.13-118.13.2.el7uek.x86_64 #2 SMP Wed Oct 5 11:03:41 PDT 2016 x86_64 x86_64 x86_64 GNU/Linux | ||
| nginx -V: |
nginx version: nginx/1.11.10 (nginx-plus-r12-p2)
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) built with OpenSSL 1.0.1e-fips 11 Feb 2013 TLS SNI support enabled configure arguments: --build=nginx-plus-r12-p2 --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_jwt_module --with-http_auth_request_module --with-http_dav_module --with-http_f4f_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_hls_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_session_log_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie' |
||
Description
nginx version: nginx/1.11.10 (nginx-plus-r12-p2)
I have used default config file from the NGINX website (but without two way auth) in order to load balance between four upstream app servers over SSL with the the proxy_ssl_verify set to on. Yes, All upstream servers have valid certificates with CN matching their hostnames, and CA has been placed on the NGINX server with proper permissions and set in the nginx.conf file. Here are my findings (removed the rest of the config as it is not relevant):
OK - Works with proxy_ssl_verify off:
stream {
upstream upstream_appsrv{
server serverA.domain.com:443;
}
server {
listen 8443;
proxy_pass https://upstream_appsrv;
(...)
proxy_ssl off;
proxy_ssl_verify off;
}
}
NOT OK - As soon as I set proxy_ssl_verify on + proxy_ssl on:
stream {
upstream upstream_appsrv{
server serverA.domain.com:443;
}
server {
listen 8443;
proxy_pass https://upstream_appsrv;
(...)
proxy_ssl on;
proxy_ssl_verify on;
}
}
NGINX started throwing errors about upstream SSL certificate not matching the backend:2 upstream SSL certificate does not match "serverB.domain.com" while SSL handshaking to upstream...Lets change some things and go to the third step.
OK! - Now everything works fine:
stream {
upstream serverA.domain.com{
server serverA.domain.com:443;
}
server {
listen 8443;
proxy_pass https://serverA.domain.com;
(...)
proxy_ssl on;
proxy_ssl_verify on;
}
}
Turns out, that not only the server definition has to match the CN used in the certificate on the upstream server (obviously) but also the upstream upstream_appsrv needs to match the CN! Ok so next step, lets add another upstream server:
NOT OK...
stream {
upstream serverA.domain.com{
server serverB.domain.com:443;
}
server {
listen 8443;
proxy_pass https://serverA.domain.com;
(...)
proxy_ssl on;
proxy_ssl_verify on;
}
}
As soon as NGINX connects to the serverB we start seeing same errors again... Even though serverB has got proper CN set matching it's hostname (and is using same CA as serverA) it turns out that NGINX tries to match the CN against the upstream definition. And that's the issue I am trying to resolve.
Change History (5)
comment:1 by , 7 years ago
comment:2 by , 7 years ago
| Resolution: | → invalid |
|---|---|
| Status: | new → closed |
Hello,
this is a bug tracking system for nginx open-source. It looks like you are using nginx-plus. In this case feel free to ask plus-support@… for support.
Thanks,
Maxim Konovalov
comment:3 by , 7 years ago
| Resolution: | invalid |
|---|---|
| Status: | closed → reopened |
Are you saying this bug does not exist in nginx (not plus) ?
comment:4 by , 7 years ago
Not really. I am saying that there is a dedicated support channel for nginx-plus customers w/ tracking, specific SLA and support quality assurance.
Thanks,
Maxim
comment:5 by , 7 years ago
| Resolution: | → invalid |
|---|---|
| Status: | reopened → closed |
This behaviour is not a bug. The proxy_pass directive verifies the name it was said to connect to. It knows nothing about how this name was resolved to addresses, much like when you type the address in a browser. All addresses the name resolves to, either via system name resolution, or via upstream{} blocks, are expected to present the certificate which matches the name.
At some extent, you may fine-tune the name nginx will verify using the proxy_ssl_name directive. It can be used if the name of the server you are requesting is different from the one written in proxy_pass, for example, when also using proxy_set_header Host .... This still won't allow to do different checks for different upstream servers, and not expected to. All the upstream servers are expected to be able to return the certificate for the name in question. Much like all the upstream servers are expected to answer requests to the same virtual host.
To resolve this, you have to reconfigure all your backend servers to return certificates with one name, much like you do when publishing a single DNS record with multiple IP addresses to use it in browsers. For example, configure nginx to request two backend servers for backend.example.com:
upstream backend.example.com {
server backend01.example.com:443;
server backend02.example.com:443;
}
server {
proxy_pass https://backend.example.com;
proxy_ssl_verify on;
proxy_ssl_trusted_certificate ...
}
and use a certificate for backend.example.com on both servers.
Seems like this module needs to support: