Opened 7 years ago

Closed 21 months ago

#1234 closed enhancement (fixed)

"proxy_ssl_trusted_certificate" is loaded into memory many times when defined in "http" context.

Reported by: zrice57@… Owned by:
Priority: minor Milestone:
Component: other Version: 1.10.x
Keywords: Memory Proxy_SSL Cc:
uname -a: Linux REDACTED_HOSTNAME 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.10.3
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-file-aio --with-threads --with-ipv6 --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_ssl_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

Description

When enabling proxy_ssl_verify we set the proxy_ssl_trusted_certificate to the default system bundle: /etc/pki/tls/cert.pem.

This cert bundle is about 250 Kilobytes.

We have thousands of server blocks defined and all of them have a value for proxy_ssl_name defined because we use IP addresses to define upstream servers.

It seems that the cert bundle is loaded many thousands of times into memory (once for each upstream?).

Nginx typically uses ~2GB of memory, but enabling proxy_ssl_verify causes it to expand to 8GB (the system max) before being killed by the kernel.

If the proxy_ssl_trusted_certificate is changed to a single cert, the memory consumed by nginx is seemingly normal again.

Change History (6)

comment:1 by Maxim Dounin, 7 years ago

Type: defectenhancement

Each location with proxy_pass to https creates it's own SSL context with it's own settings, including trusted certificates if defined, so the observed behaviour is expected. Optimizing this is possible to some extent (as long as all proxy_ssl_* settings are identical), though needs someone to do the work.

If you are interested in optimization of configurations with thousands of server blocks, consider updating to at least 1.11.6, which already has various optimizations in place (7e5199f172fb, 93121485c39b, 63991ab67b3a, 20eb4587225b).

comment:2 by zrice57@…, 7 years ago

In our case proxy_ssl_name is different for every location with a proxy_pass. We could probably re-engineer our setup so that they can be the same.

comment:3 by Maxim Dounin, 7 years ago

The proxy_ssl_name directive do not affects SSL context, only a particular location configuration, so it shouldn't make a difference. Problematic directives are ones passed to OpenSSL when creating SSL context - certificates, ciphers, protocols, and so on. This includes almost all proxy_ssl_* directives except a few.

comment:4 by zrice57@…, 7 years ago

Thanks for the info!

comment:5 by Maxim Dounin <mdounin@…>, 21 months ago

In 8053:9d98d524bd02/nginx:

Upstream: optimized use of SSL contexts (ticket #1234).

To ensure optimal use of memory, SSL contexts for proxying are now
inherited from previous levels as long as relevant proxy_ssl_* directives
are not redefined.

Further, when no proxy_ssl_* directives are redefined in a server block,
we now preserve plcf->upstream.ssl in the "http" section configuration
to inherit it to all servers.

Similar changes made in uwsgi, grpc, and stream proxy.

comment:6 by Sergey Kandaurov, 21 months ago

Resolution: fixed
Status: newclosed

Fix committed.

Note: See TracTickets for help on using tickets.