Opened 9 years ago

Closed 5 years ago

#714 closed defect (duplicate)

Writing connection leak while SPDY enabled

Reported by: Ivan Ivanov Owned by:
Priority: minor Milestone:
Component: nginx-core Version: 1.6.x
Keywords: spdy Cc:
uname -a: Linux nginx 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: [root@nginx ~]# /usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.6.2
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC)
TLS SNI support enabled
configure arguments: --add-module=../ngx_pagespeed-master --with-debug --with-http_ssl_module --with-http_sub_module --with-http_stub_status_module --add-module=../echo-nginx-module-0.51 --with-http_spdy_module
[root@nginx ~]#

Description

I'm trying to use SPDY module, but when I enable this into my config
listen 443 ssl spdy;
The number of Writing connections start to leak.

Attachments (10)

graph_image (1).png (23.8 KB ) - added by Ivan Ivanov 9 years ago.
graph_image.png (21.9 KB ) - added by Ivan Ivanov 9 years ago.
nginx-without-3rd-party-1.png (23.4 KB ) - added by Ivan Ivanov 9 years ago.
nginx-without-3rd-party.png (23.5 KB ) - added by Ivan Ivanov 9 years ago.
nginx-1.7.10-7-day.png (95.1 KB ) - added by Hendrik Horeis 9 years ago.
7 day increase of writing connections with spdy
nginx_status-week.png (26.7 KB ) - added by Ondrej Brbalc 9 years ago.
Leaking writing connections in 1.8
nginx_status-day.png (27.6 KB ) - added by Ondrej Brbalc 9 years ago.
nginx_status-week.2.png (22.5 KB ) - added by ron.gerrits@… 8 years ago.
nginxcrashspdy
nginx_status-year.png (29.7 KB ) - added by ron.gerrits@… 8 years ago.
Hanging wrting connections are back
nginx_leak.jpg (94.2 KB ) - added by Nar 5 years ago.

Download all attachments as: .zip

Change History (33)

by Ivan Ivanov, 9 years ago

Attachment: graph_image (1).png added

by Ivan Ivanov, 9 years ago

Attachment: graph_image.png added

comment:1 by Valentin V. Bartenev, 9 years ago

Could you reproduce it without 3rd-party modules?

comment:2 by Ivan Ivanov, 9 years ago

Yes, the situation is same.
I compile nginx without 3rd-party modules, but "Writing" connections grow slowly.
Let me know if you need any other information.

comment:3 by Ivan Ivanov, 9 years ago

[root@nginx1 ~]# ps x | grep nginx

8836 ? Ss 0:00 nginx: master process /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf

[root@nginx1 ~]#
[root@nginx1 ~]# /usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.6.2
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC)
TLS SNI support enabled
configure arguments: --with-debug --with-http_ssl_module --with-http_sub_module --with-http_stub_status_module --with-http_spdy_module
[root@nginx1 ~]#
[root@nginx1 ~]# stat /usr/local/nginx/sbin/nginx

File: `/usr/local/nginx/sbin/nginx'
Size: 5747928 Blocks: 11232 IO Block: 4096 regular file

Device: fd00h/64768d Inode: 1053772 Links: 1
Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2015-02-06 09:34:24.078330934 +0200
Modify: 2015-02-06 09:34:11.094331557 +0200
Change: 2015-02-06 09:34:11.094331557 +0200
[root@nginx1 ~]#

Will attach new cacti graphs:
nginx-without-3rd-party.png
nginx-without-3rd-party-1.png

by Ivan Ivanov, 9 years ago

by Ivan Ivanov, 9 years ago

Attachment: nginx-without-3rd-party.png added

comment:4 by Maxim Dounin, 9 years ago

It might be a good idea to try nginx 1.7.9 instead of 1.6.2.

comment:5 by Ivan Ivanov, 9 years ago

I will try next week and will write the results.

comment:6 by Ivan Ivanov, 9 years ago

Hello,
I upgraded to version 1.7.9 and problem does not occur for several hours.
This version probably is not affected.
If the problem occurs a later, I will open new ticket.
Thank you for cooperation.

comment:7 by Hendrik Horeis, 9 years ago

Hi guys,

I just feel free to leave an update here and to confirm that I am also experiencing this behavior with the latest mainline version 1.7.10. See latest attachment too see the increase of writing connections over a 7 day period. No negative side effects seen so far.

nginx version: nginx/1.7.10
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-mail --with-mail_ssl_module --with-file-aio --with-http_spdy_module --with-http_geoip_module --with-ipv6 --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic'

2.6.32-504.1.3.el6.x86_64 #1 SMP Tue Nov 11 17:57:25 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

by Hendrik Horeis, 9 years ago

Attachment: nginx-1.7.10-7-day.png added

7 day increase of writing connections with spdy

comment:8 by Valentin V. Bartenev, 9 years ago

Could you provide your configuration? Do you have anything in the error log?

comment:9 by Hendrik Horeis, 9 years ago

Hi Valentin,

phew, the set of configs is kind of massive. lots of individual configs in conf.d. error log is clean beside some upstream timeouts, no spdy errors at all. are you looking for anything specific?

by Ondrej Brbalc, 9 years ago

Attachment: nginx_status-week.png added

Leaking writing connections in 1.8

comment:10 by Ondrej Brbalc, 9 years ago

I just want to report this issue with the latest stable version 1.8.0

nginx version: nginx/1.8.0
built by gcc 4.7.2 (Debian 4.7.2-5)
built with OpenSSL 1.0.1e 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-mail --with-mail_ssl_module --with-file-aio --with-http_spdy_module --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,--as-needed' --with-ipv6

$ uname -a
Linux cl-lb1 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u2 x86_64 GNU/Linux

We have turned spdy for our static content which makes about 70% of traffic. It was server by plain http before (and it is pretty amazing, that it works!). So we are not sure whether this is spdy issue or https. But we used to have some https traffic before (less then 1%) and writing connections were not leaking.

See the image:
Leaking writing connections in 1.8
The sudden drop is a restart. Just before switching spdy there was an outage of internet connection. So ignore the drop of waiting connections.

The main question is - before it gets fixed should we ignore the value leaking or do preventive restarts?

comment:11 by Valentin V. Bartenev, 9 years ago

Reloading the configuration should be enough. But please note, that it's not something easy to find and fix, since it's not reproducible with simple configs or even in a test environment.

So I'm kindly asking again everybody who experiences this issue to provide as much information as possible: configuration, debug logs, etc. If someone can provide steps to reproduce, then I'll fix it very soon.

Last edited 9 years ago by Valentin V. Bartenev (previous) (diff)

by Ondrej Brbalc, 9 years ago

Attachment: nginx_status-day.png added

comment:12 by Ondrej Brbalc, 9 years ago

Restart is required, reload does not help visually. I was not sure, if it was https or spdy, so now I confirmed this to happen with spdy only. We have rather simple config files, may I mail them somewhere? On this chart it is perhaps interesting to see how it behaved when traffic was decreasing. It is also visible, that active connections are not following the waiting connections, when writing connections is leaking.


comment:13 by Valentin V. Bartenev, 9 years ago

You can send configuration to my mail: vbart@…

comment:14 by Maxim Dounin, 9 years ago

Keywords: spdy added

comment:15 by Commenter123@…, 8 years ago

Use HTTP2.

comment:16 by ron.gerrits@…, 8 years ago

Same problem here. Reloading configuration does not help, I have to restart nginx to fix the problem each time. My server crashed on this problem. SPDY is enabled. This is my config:

server {
    listen       443 default_server spdy;
    server_name  -snip-;

    ssl                  on;
    ssl_certificate      /etc/nginx/ssl.d/snip.chained.crt;
    ssl_certificate_key  /etc/nginx/ssl.d/snip.key;

    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/nginx/ssl.d/ca.bundle.ocsp.pem;

    ssl_session_tickets on;
    ssl_session_timeout  5m;
    ssl_session_cache shared:SSL:5M;

    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS;
    ssl_prefer_server_ciphers   on;
    ssl_dhparam /etc/nginx/ssl.d/dhparam.pem;

    root   /snip/;
    index index.php index.html index.htm;

    open_file_cache          max=2000 inactive=20s;
    open_file_cache_valid    60s;
    open_file_cache_min_uses 5;
    open_file_cache_errors   off;

 uname -a
Linux snip 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Sep 15 15:05:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
nginx -V
nginx version: nginx/1.6.3
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC)
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-file-aio --with-ipv6 --with-http_ssl_module --with-http_spdy_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module --with-http_image_filter_module --with-http_geoip_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_stub_status_module --with-http_perl_module --with-mail --with-mail_ssl_module --with-pcre --with-pcre-jit --with-google_perftools_module --with-debug --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E'
 yum info nginx
Installed Packages
Name        : nginx
Arch        : x86_64
Epoch       : 1
Version     : 1.6.3
Release     : 8.el7
Size        : 1.4 M
Repo        : installed
From repo   : epel
Summary     : A high performance web server and reverse proxy server
URL         : http://nginx.org/
License     : BSD
Description : Nginx is a web server and a reverse proxy server for HTTP, SMTP, POP3 and
            : IMAP protocols, with a strong focus on high concurrency, performance and low
            : memory usage.

And my munin graph in the attachment.
https://trac.nginx.org/nginx/raw-attachment/ticket/714/nginx_status-week.2.png

I've got SELinux enabled on my server.

Last edited 8 years ago by ron.gerrits@… (previous) (diff)

by ron.gerrits@…, 8 years ago

Attachment: nginx_status-week.2.png added

nginxcrashspdy

comment:17 by Maxim Dounin, 8 years ago

This looks like a duplicate of ticket #626, fixed in 1.9.12. Please try nginx 1.9.12 to see if it helps.

comment:18 by ron.gerrits@…, 8 years ago

Yes it does seem like a duplicate of #626. I'm seeing these errors in my log when reloading:

2016/03/17 09:43:47 [alert] 11819#0: *43809 open socket #49 left in connection 4
2016/03/17 09:43:47 [alert] 11819#0: *43920 open socket #45 left in connection 9
2016/03/17 09:43:47 [alert] 11819#0: *49839 open socket #39 left in connection 17
2016/03/17 09:43:47 [alert] 11819#0: *44130 open socket #21 left in connection 29
2016/03/17 09:43:47 [alert] 11819#0: *39756 open socket #55 left in connection 34
2016/03/17 09:43:47 [alert] 11819#0: *41352 open socket #15 left in connection 44
2016/03/17 09:43:47 [alert] 11819#0: *40153 open socket #14 left in connection 58
2016/03/17 09:43:47 [alert] 11819#0: *44279 open socket #48 left in connection 64
2016/03/17 09:43:47 [alert] 11819#0: *69858 open socket #30 left in connection 73
2016/03/17 09:43:47 [alert] 11819#0: *40706 open socket #56 left in connection 77
2016/03/17 09:43:47 [alert] 11819#0: aborting

Shouldn't the EPEL version of Nginx be patched or something?

comment:19 by Maxim Dounin, 8 years ago

Resolution: duplicate
Status: newclosed

You can install nginx 1.9.12 from sources or using our packages, see http://nginx.org/en/download.html. Thanks for testing, closing this.

comment:20 by ron.gerrits@…, 8 years ago

The bug has re-appeared since the last update of nginx. It was gone for some time, but reappeared in week 33/34.

Error log is filled with these:

2016/09/09 12:07:37 [alert] 22910#22910: *1682525 open socket #65 left in connection 121
2016/09/09 12:07:37 [alert] 22910#22910: *2481905 open socket #24 left in connection 142
2016/09/09 12:07:37 [alert] 22910#22910: *834242 open socket #43 left in connection 144
2016/09/09 12:07:37 [alert] 22910#22910: *985206 open socket #12 left in connection 163
2016/09/09 12:07:37 [alert] 22910#22910: *2529918 open socket #33 left in connection 165

nginx -V

nginx version: nginx/1.10.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_perl_module=dynamic --add-dynamic-module=njs-1c50334fbea6/nginx --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --with-http_v2_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic'

uname -a

Linux snip 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Sep 15 15:05:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

And my config nowadays:

server {
    listen       443 default_server ssl http2;
    server_name  snip;

    ssl                  on;
    ssl_certificate      snip.crt;
    ssl_certificate_key  snip.key;

    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate snip.pem;

    ssl_session_tickets on;
    ssl_session_timeout  5m;
    ssl_session_cache shared:SSL:5M;

    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS;
    ssl_prefer_server_ciphers   on;
    ssl_dhparam snip.pem;

(...)

https://trac.nginx.org/nginx/raw-attachment/ticket/714/nginx_status-year.png

Last edited 8 years ago by ron.gerrits@… (previous) (diff)

by ron.gerrits@…, 8 years ago

Attachment: nginx_status-year.png added

Hanging wrting connections are back

comment:21 by Maxim Dounin, 8 years ago

There is at least one connection leak in HTTP/2 as fixed in nginx 1.11.3, and the fix is not yet available in 1.10.x. Please try 1.11.x instead.

by Nar, 5 years ago

Attachment: nginx_leak.jpg added

comment:22 by Nar, 5 years ago

Resolution: duplicate
Status: closedreopened

Have same issue with nginx 1.15.7


uname -a

Linux CentOS-72-64-minimal 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6 23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

nginx -V

nginx version: nginx/1.15.7
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-p                                                                                                                                                             ath=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=                                                                                                                                                             /var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/v                                                                                                                                                             ar/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/v                                                                                                                                                             ar/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --                                                                                                                                                             http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var                                                                                                                                                             /cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=                                                                                                                                                             nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_add                                                                                                                                                             ition_module --with-http_auth_request_module --with-http_dav_module --with-http_                                                                                                                                                             flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_                                                                                                                                                             mp4_module --with-http_random_index_module --with-http_realip_module --with-http                                                                                                                                                             _secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_                                                                                                                                                             stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --wi                                                                                                                                                             th-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_m                                                                                                                                                             odule --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_                                                                                                                                                             FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4                                                                                                                                                              -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -W                                                                                                                                                             l,-z,now -pie'

comment:23 by Maxim Dounin, 5 years ago

Resolution: duplicate
Status: reopenedclosed

For most recent tickets about HTTP/2 issues, please refer to #1610, #1163, #1650. Reopening this ancient SPDY ticket hardly makes sense.

Note: See TracTickets for help on using tickets.