Opened 9 months ago

Last modified 8 months ago

#1234 new enhancement

"proxy_ssl_trusted_certificate" is loaded into memory many times when defined in "http" context.

Reported by: zrice57@… Owned by:
Priority: minor Milestone:
Component: other Version: 1.10.x
Keywords: Memory Proxy_SSL Cc:
Sensitive: no
uname -a: Linux REDACTED_HOSTNAME 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.10.3 built by gcc 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) built with OpenSSL 1.0.1e-fips 11 Feb 2013 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-file-aio --with-threads --with-ipv6 --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_ssl_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

Description

When enabling proxy_ssl_verify we set the proxy_ssl_trusted_certificate to the default system bundle: /etc/pki/tls/cert.pem.

This cert bundle is about 250 Kilobytes.

We have thousands of server blocks defined and all of them have a value for proxy_ssl_name defined because we use IP addresses to define upstream servers.

It seems that the cert bundle is loaded many thousands of times into memory (once for each upstream?).

Nginx typically uses ~2GB of memory, but enabling proxy_ssl_verify causes it to expand to 8GB (the system max) before being killed by the kernel.

If the proxy_ssl_trusted_certificate is changed to a single cert, the memory consumed by nginx is seemingly normal again.

Change History (4)

comment:1 Changed 9 months ago by mdounin

  • Type changed from defect to enhancement

Each location with proxy_pass to https creates it's own SSL context with it's own settings, including trusted certificates if defined, so the observed behaviour is expected. Optimizing this is possible to some extent (as long as all proxy_ssl_* settings are identical), though needs someone to do the work.

If you are interested in optimization of configurations with thousands of server blocks, consider updating to at least 1.11.6, which already has various optimizations in place (7e5199f172fb, 93121485c39b, 63991ab67b3a, 20eb4587225b).

comment:2 Changed 9 months ago by zrice57@…

In our case proxy_ssl_name is different for every location with a proxy_pass. We could probably re-engineer our setup so that they can be the same.

comment:3 Changed 8 months ago by mdounin

The proxy_ssl_name directive do not affects SSL context, only a particular location configuration, so it shouldn't make a difference. Problematic directives are ones passed to OpenSSL when creating SSL context - certificates, ciphers, protocols, and so on. This includes almost all proxy_ssl_* directives except a few.

comment:4 Changed 8 months ago by zrice57@…

Thanks for the info!

Note: See TracTickets for help on using tickets.