Writing connection leak while SPDY enabled

[Ivan Ivanov]

I'm trying to use SPDY module, but when I enable this into my config
listen 443 ssl spdy;
The number of Writing connections start to leak.

[Valentin V. Bartenev]

Could you reproduce it without 3rd-party modules?

[Ivan Ivanov]

Yes, the situation is same.
I compile nginx without 3rd-party modules, but "Writing" connections grow slowly.
Let me know if you need any other information.

[Ivan Ivanov]

[root@nginx1 ~]# ps x | grep nginx

8836 ? Ss 0:00 nginx: master process /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf

[root@nginx1 ~]#
[root@nginx1 ~]# /usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.6.2
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC)
TLS SNI support enabled
configure arguments: --with-debug --with-http_ssl_module --with-http_sub_module --with-http_stub_status_module --with-http_spdy_module
[root@nginx1 ~]#
[root@nginx1 ~]# stat /usr/local/nginx/sbin/nginx

File: `/usr/local/nginx/sbin/nginx'
Size: 5747928 Blocks: 11232 IO Block: 4096 regular file

Device: fd00h/64768d Inode: 1053772 Links: 1
Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2015-02-06 09:34:24.078330934 +0200
Modify: 2015-02-06 09:34:11.094331557 +0200
Change: 2015-02-06 09:34:11.094331557 +0200
[root@nginx1 ~]#

Will attach new cacti graphs:
nginx-without-3rd-party.png
nginx-without-3rd-party-1.png

[Maxim Dounin]

It might be a good idea to try nginx 1.7.9 instead of 1.6.2.

[Ivan Ivanov]

I will try next week and will write the results.

[Ivan Ivanov]

Hello,
I upgraded to version 1.7.9 and problem does not occur for several hours.
This version probably is not affected.
If the problem occurs a later, I will open new ticket.
Thank you for cooperation.

[Hendrik Horeis]

Hi guys,

I just feel free to leave an update here and to confirm that I am also experiencing this behavior with the latest mainline version 1.7.10. See latest attachment too see the increase of writing connections over a 7 day period. No negative side effects seen so far.

nginx version: nginx/1.7.10
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-mail --with-mail_ssl_module --with-file-aio --with-http_spdy_module --with-http_geoip_module --with-ipv6 --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic'

2.6.32-504.1.3.el6.x86_64 #1 SMP Tue Nov 11 17:57:25 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

[Hendrik Horeis]

7 day increase of writing connections with spdy

[Valentin V. Bartenev]

Could you provide your configuration? Do you have anything in the error log?

[Hendrik Horeis]

Hi Valentin,

phew, the set of configs is kind of massive. lots of individual configs in conf.d. error log is clean beside some upstream timeouts, no spdy errors at all. are you looking for anything specific?

[Ondrej Brbalc]

Leaking writing connections in 1.8

[Ondrej Brbalc]

I just want to report this issue with the latest stable version 1.8.0

nginx version: nginx/1.8.0
built by gcc 4.7.2 (Debian 4.7.2-5)
built with OpenSSL 1.0.1e 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-mail --with-mail_ssl_module --with-file-aio --with-http_spdy_module --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,--as-needed' --with-ipv6

$ uname -a
Linux cl-lb1 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u2 x86_64 GNU/Linux

We have turned spdy for our static content which makes about 70% of traffic. It was server by plain http before (and it is pretty amazing, that it works!). So we are not sure whether this is spdy issue or https. But we used to have some https traffic before (less then 1%) and writing connections were not leaking.

See the image:

The sudden drop is a restart. Just before switching spdy there was an outage of internet connection. So ignore the drop of waiting connections.

The main question is - before it gets fixed should we ignore the value leaking or do preventive restarts?

[Valentin V. Bartenev]

Reloading the configuration should be enough. But please note, that it's not something easy to find and fix, since it's not reproducible with simple configs or even in a test environment.

So I'm kindly asking again everybody who experiences this issue to provide as much information as possible: configuration, debug logs, etc. If someone can provide steps to reproduce, then I'll fix it very soon.

[Ondrej Brbalc]

Restart is required, reload does not help visually. I was not sure, if it was https or spdy, so now I confirmed this to happen with spdy only. We have rather simple config files, may I mail them somewhere? On this chart it is perhaps interesting to see how it behaved when traffic was decreasing. It is also visible, that active connections are not following the waiting connections, when writing connections is leaking.

[Valentin V. Bartenev]

You can send configuration to my mail: vbart@…

[Commenter123@…]

Use HTTP2.

[ron.gerrits@…]

Same problem here. Reloading configuration does not help, I have to restart nginx to fix the problem each time. My server crashed on this problem. SPDY is enabled. This is my config:

server {
    listen       443 default_server spdy;
    server_name  -snip-;

    ssl                  on;
    ssl_certificate      /etc/nginx/ssl.d/snip.chained.crt;
    ssl_certificate_key  /etc/nginx/ssl.d/snip.key;

    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/nginx/ssl.d/ca.bundle.ocsp.pem;

    ssl_session_tickets on;
    ssl_session_timeout  5m;
    ssl_session_cache shared:SSL:5M;

    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS;
    ssl_prefer_server_ciphers   on;
    ssl_dhparam /etc/nginx/ssl.d/dhparam.pem;

    root   /snip/;
    index index.php index.html index.htm;

    open_file_cache          max=2000 inactive=20s;
    open_file_cache_valid    60s;
    open_file_cache_min_uses 5;
    open_file_cache_errors   off;

 uname -a
Linux snip 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Sep 15 15:05:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

nginx -V
nginx version: nginx/1.6.3
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC)
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-file-aio --with-ipv6 --with-http_ssl_module --with-http_spdy_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module --with-http_image_filter_module --with-http_geoip_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_stub_status_module --with-http_perl_module --with-mail --with-mail_ssl_module --with-pcre --with-pcre-jit --with-google_perftools_module --with-debug --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E'

 yum info nginx
Installed Packages
Name        : nginx
Arch        : x86_64
Epoch       : 1
Version     : 1.6.3
Release     : 8.el7
Size        : 1.4 M
Repo        : installed
From repo   : epel
Summary     : A high performance web server and reverse proxy server
URL         : http://nginx.org/
License     : BSD
Description : Nginx is a web server and a reverse proxy server for HTTP, SMTP, POP3 and
            : IMAP protocols, with a strong focus on high concurrency, performance and low
            : memory usage.

And my munin graph in the attachment.

[ron.gerrits@…]

nginxcrashspdy

[Maxim Dounin]

This looks like a duplicate of ticket #626, fixed in 1.9.12. Please try nginx 1.9.12 to see if it helps.

[ron.gerrits@…]

Yes it does seem like a duplicate of #626. I'm seeing these errors in my log when reloading:

2016/03/17 09:43:47 [alert] 11819#0: *43809 open socket #49 left in connection 4
2016/03/17 09:43:47 [alert] 11819#0: *43920 open socket #45 left in connection 9
2016/03/17 09:43:47 [alert] 11819#0: *49839 open socket #39 left in connection 17
2016/03/17 09:43:47 [alert] 11819#0: *44130 open socket #21 left in connection 29
2016/03/17 09:43:47 [alert] 11819#0: *39756 open socket #55 left in connection 34
2016/03/17 09:43:47 [alert] 11819#0: *41352 open socket #15 left in connection 44
2016/03/17 09:43:47 [alert] 11819#0: *40153 open socket #14 left in connection 58
2016/03/17 09:43:47 [alert] 11819#0: *44279 open socket #48 left in connection 64
2016/03/17 09:43:47 [alert] 11819#0: *69858 open socket #30 left in connection 73
2016/03/17 09:43:47 [alert] 11819#0: *40706 open socket #56 left in connection 77
2016/03/17 09:43:47 [alert] 11819#0: aborting

Shouldn't the EPEL version of Nginx be patched or something?

[Maxim Dounin]

You can install nginx 1.9.12 from sources or using our packages, see ​http://nginx.org/en/download.html. Thanks for testing, closing this.

[ron.gerrits@…]

The bug has re-appeared since the last update of nginx. It was gone for some time, but reappeared in week 33/34.

Error log is filled with these:

2016/09/09 12:07:37 [alert] 22910#22910: *1682525 open socket #65 left in connection 121
2016/09/09 12:07:37 [alert] 22910#22910: *2481905 open socket #24 left in connection 142
2016/09/09 12:07:37 [alert] 22910#22910: *834242 open socket #43 left in connection 144
2016/09/09 12:07:37 [alert] 22910#22910: *985206 open socket #12 left in connection 163
2016/09/09 12:07:37 [alert] 22910#22910: *2529918 open socket #33 left in connection 165

nginx -V

nginx version: nginx/1.10.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_perl_module=dynamic --add-dynamic-module=njs-1c50334fbea6/nginx --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --with-http_v2_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic'

uname -a

Linux snip 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Sep 15 15:05:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

And my config nowadays:

server {
    listen       443 default_server ssl http2;
    server_name  snip;

    ssl                  on;
    ssl_certificate      snip.crt;
    ssl_certificate_key  snip.key;

    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate snip.pem;

    ssl_session_tickets on;
    ssl_session_timeout  5m;
    ssl_session_cache shared:SSL:5M;

    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS;
    ssl_prefer_server_ciphers   on;
    ssl_dhparam snip.pem;

(...)

[ron.gerrits@…]

Hanging wrting connections are back

[Maxim Dounin]

There is at least one connection leak in HTTP/2 as fixed in nginx 1.11.3, and the fix is not yet available in 1.10.x. Please try 1.11.x instead.

[Nar]

Have same issue with nginx 1.15.7

uname -a

Linux CentOS-72-64-minimal 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6 23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

nginx -V

nginx version: nginx/1.15.7
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-p                                                                                                                                                             ath=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=                                                                                                                                                             /var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/v                                                                                                                                                             ar/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/v                                                                                                                                                             ar/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --                                                                                                                                                             http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var                                                                                                                                                             /cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=                                                                                                                                                             nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_add                                                                                                                                                             ition_module --with-http_auth_request_module --with-http_dav_module --with-http_                                                                                                                                                             flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_                                                                                                                                                             mp4_module --with-http_random_index_module --with-http_realip_module --with-http                                                                                                                                                             _secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_                                                                                                                                                             stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --wi                                                                                                                                                             th-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_m                                                                                                                                                             odule --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_                                                                                                                                                             FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4                                                                                                                                                              -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -W                                                                                                                                                             l,-z,now -pie'

[Maxim Dounin]

For most recent tickets about HTTP/2 issues, please refer to #1610, #1163, #1650. Reopening this ancient SPDY ticket hardly makes sense.