Opened 8 years ago

Closed 4 months ago

#621 closed defect (fixed)

Could not allocate new session in SSL session shared cache

Reported by: Johannes Ziemke Owned by:
Priority: major Milestone:
Component: nginx-core Version: 1.7.x
Keywords: ssl Cc:
uname -a: Linux [host] 3.8.0-35-generic #52~precise1-Ubuntu SMP Thu Jan 30 17:24:40 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.7.1
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/ --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_spdy_module --with-http_sub_module --with-http_xslt_module --with-mail --with-mail_ssl_module --add-module=/build/buildd/nginx-1.7.1/debian/modules/nginx-auth-pam --add-module=/build/buildd/nginx-1.7.1/debian/modules/nginx-dav-ext-module --add-module=/build/buildd/nginx-1.7.1/debian/modules/nginx-echo --add-module=/build/buildd/nginx-1.7.1/debian/modules/nginx-upstream-fair --add-module=/build/buildd/nginx-1.7.1/debian/modules/ngx_http_substitutions_filter_module



I'm using nginx as reverse proxy in front of haproxy. I'm using this ssl_session_cache config:

ssl_session_cache   shared:SSL:10m;
ssl_session_timeout 512m;

Now I get from time to time such errors in the log:

2014-09-03T09:51:34+00:00 hostname nginx: 2014/09/03 09:51:34 [alert] 27#0: *144835271 could not allocate new session in SSL session shared cache "SSL" while SSL handshaking, client: a.b.c.d, server:

Unfortunately that error doesn't say much. Looking at the code shows that it probably failed to allocate via ngx_slab_alloc_locked():

I'll try to raise the session cache and see if it helps, but since it's just a cache I would expect only performance differences.

FWIW: The nginx is running in a docker container, to be specific: (Although I've raised the cache setting there already

Change History (5)

comment:1 by Maxim Dounin, 8 years ago

As the error says, nginx wasn't able to allocate new session in SSL shared cache. This is something that may happen if the cache is full, and nginx wasn't able to free enough space by removing a least recently used session. Use smaller session timeouts or a larger shared memory zone to avoid cache overflows.

The message is probably a bit too scary (while the situation itself is mostly harmless). It was set to "alert" level as it wasn't expected to happen in real life. It seems to happen though - especially after 5024d29354f1, with configurations assuming nginx will silently throw away old sessions if there will be not enough memory in the shared memory zone. The severity of the error should be reduced, and/or nginx should try harder (free more old sessions) before giving up.

For now you may use smaller session timeouts and/or a larger shared memory zone. This way sessions will expire and will be removed from the cache before the cache will be overflowed, and no errors will happen.

comment:2 by Maxim Dounin, 7 years ago

Status: newaccepted

comment:3 by rosenqui@…, 4 years ago

+1 on changing this log entry from alert to warning.

We've noticed this alert at busy sites with lots of clients, and found that having a mix of sessions that do/don't use client certificates makes it a lot worse.

Without a client certificate, SSL session data is about 150b. If there's a client certificate, the encoded size of the certificate is added to that, making if more like 1kb depending on the DN sizes, extensions, etc.

If the cache is full and there are no expired sessions, then when a new session using a client certificate is created, the code that tries to make space by getting rid of the oldest unexpired session won't free up anywhere near enough room if the oldest session didn't use a client certificate. Event if the oldest session did use a client certificate, its encoded size could be much smaller than the new session.

If ngx_ssl_expire_sessions was told how many bytes to free up rather than how many entries (with perhaps a cap on max # to purge), the alert messages would almost never appear.

comment:4 by Maxim Dounin <mdounin@…>, 4 months ago

In 8075:38c71f9b2293/nginx:

SSL: reduced logging of session cache failures (ticket #621).

Session cache allocations might fail as long as the new session is different
in size from the one least recently used (and freed when the first allocation
fails). In particular, it might not be possible to allocate space for
sessions with client certificates, since they are noticeably bigger than
normal sessions.

To ensure such allocation failures won't clutter logs, logging level changed
to "warn", and logging is now limited to at most one warning per second.

comment:5 by Maxim Dounin, 4 months ago

Resolution: fixed
Status: acceptedclosed

Fix committed, thanks to all involved for prodding this.

Note: See TracTickets for help on using tickets.