#1307 closed defect (invalid)

proxy_ssl_verify fails with multiple upstreams

Reported by: wheelq@… Owned by:
Priority: blocker Milestone:
Component: nginx-module Version: 1.11.x
Keywords: ssl Cc:
uname -a: Linux xxxxx 3.8.13-118.13.2.el7uek.x86_64 #2 SMP Wed Oct 5 11:03:41 PDT 2016 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.11.10 (nginx-plus-r12-p2) built by gcc 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) built with OpenSSL 1.0.1e-fips 11 Feb 2013 TLS SNI support enabled configure arguments: --build=nginx-plus-r12-p2 --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_jwt_module --with-http_auth_request_module --with-http_dav_module --with-http_f4f_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_hls_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_session_log_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

Description

nginx version: nginx/1.11.10 (nginx-plus-r12-p2)

I have used default config file from the NGINX website (but without two way auth) in order to load balance between four upstream app servers over SSL with the the proxy_ssl_verify set to on. Yes, All upstream servers have valid certificates with CN matching their hostnames, and CA has been placed on the NGINX server with proper permissions and set in the nginx.conf file. Here are my findings (removed the rest of the config as it is not relevant):

OK - Works with proxy_ssl_verify off:
stream {

upstream upstream_appsrv{

server serverA.domain.com:443;

}
server {

listen 8443;
proxy_pass https://upstream_appsrv;

(...)

proxy_ssl off;
proxy_ssl_verify off;

}

}
NOT OK - As soon as I set proxy_ssl_verify on + proxy_ssl on:
stream {

upstream upstream_appsrv{

server serverA.domain.com:443;

}
server {

listen 8443;
proxy_pass https://upstream_appsrv;
(...)
proxy_ssl on;
proxy_ssl_verify on;

}

}
NGINX started throwing errors about upstream SSL certificate not matching the backend:2 upstream SSL certificate does not match "serverB.domain.com" while SSL handshaking to upstream...Lets change some things and go to the third step.

OK! - Now everything works fine:
stream {

upstream serverA.domain.com{

server serverA.domain.com:443;

}
server {

listen 8443;
proxy_pass https://serverA.domain.com;
(...)
proxy_ssl on;
proxy_ssl_verify on;

}

}
Turns out, that not only the server definition has to match the CN used in the certificate on the upstream server (obviously) but also the upstream upstream_appsrv needs to match the CN! Ok so next step, lets add another upstream server:

NOT OK...
stream {

upstream serverA.domain.com{

server serverB.domain.com:443;

}
server {

listen 8443;
proxy_pass https://serverA.domain.com;
(...)
proxy_ssl on;
proxy_ssl_verify on;

}

}
As soon as NGINX connects to the serverB we start seeing same errors again... Even though serverB has got proper CN set matching it's hostname (and is using same CA as serverA) it turns out that NGINX tries to match the CN against the upstream definition. And that's the issue I am trying to resolve.

Change History (5)

comment:1 Changed 21 months ago by wheelq@…

Seems like this module needs to support:

  • proxy_ssl_verify_ca to verify just the CA, not CN
  • or proxy_ssl_verify_cn with options ON|OFF
  • should not be checking the upstream alias as CN

comment:2 Changed 21 months ago by maxim

  • Resolution set to invalid
  • Status changed from new to closed

Hello,

this is a bug tracking system for nginx open-source. It looks like you are using nginx-plus. In this case feel free to ask plus-support@… for support.

Thanks,

Maxim Konovalov

comment:3 Changed 21 months ago by wheelq@…

  • Resolution invalid deleted
  • Status changed from closed to reopened

Are you saying this bug does not exist in nginx (not plus) ?

comment:4 Changed 21 months ago by maxim

Not really. I am saying that there is a dedicated support channel for nginx-plus customers w/ tracking, specific SLA and support quality assurance.

Thanks,

Maxim

comment:5 Changed 21 months ago by mdounin

  • Resolution set to invalid
  • Status changed from reopened to closed

This behaviour is not a bug. The proxy_pass directive verifies the name it was said to connect to. It knows nothing about how this name was resolved to addresses, much like when you type the address in a browser. All addresses the name resolves to, either via system name resolution, or via upstream{} blocks, are expected to present the certificate which matches the name.

At some extent, you may fine-tune the name nginx will verify using the proxy_ssl_name directive. It can be used if the name of the server you are requesting is different from the one written in proxy_pass, for example, when also using proxy_set_header Host .... This still won't allow to do different checks for different upstream servers, and not expected to. All the upstream servers are expected to be able to return the certificate for the name in question. Much like all the upstream servers are expected to answer requests to the same virtual host.

To resolve this, you have to reconfigure all your backend servers to return certificates with one name, much like you do when publishing a single DNS record with multiple IP addresses to use it in browsers. For example, configure nginx to request two backend servers for backend.example.com:

upstream backend.example.com {
    server backend01.example.com:443;
    server backend02.example.com:443;
}

server {
    proxy_pass https://backend.example.com;
    proxy_ssl_verify on;
    proxy_ssl_trusted_certificate ...
}

and use a certificate for backend.example.com on both servers.

Note: See TracTickets for help on using tickets.