Opened 10 years ago
Closed 6 years ago
#714 closed defect (duplicate)
Writing connection leak while SPDY enabled
Reported by: | Ivan Ivanov | Owned by: | |
---|---|---|---|
Priority: | minor | Milestone: | |
Component: | nginx-core | Version: | 1.6.x |
Keywords: | spdy | Cc: | |
uname -a: | Linux nginx 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | ||
nginx -V: |
[root@nginx ~]# /usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.6.2 built by gcc 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) TLS SNI support enabled configure arguments: --add-module=../ngx_pagespeed-master --with-debug --with-http_ssl_module --with-http_sub_module --with-http_stub_status_module --add-module=../echo-nginx-module-0.51 --with-http_spdy_module [root@nginx ~]# |
Description
I'm trying to use SPDY module, but when I enable this into my config
listen 443 ssl spdy;
The number of Writing connections start to leak.
Attachments (10)
Change History (33)
by , 10 years ago
Attachment: | graph_image (1).png added |
---|
by , 10 years ago
Attachment: | graph_image.png added |
---|
comment:1 by , 10 years ago
comment:2 by , 10 years ago
Yes, the situation is same.
I compile nginx without 3rd-party modules, but "Writing" connections grow slowly.
Let me know if you need any other information.
comment:3 by , 10 years ago
[root@nginx1 ~]# ps x | grep nginx
8836 ? Ss 0:00 nginx: master process /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
[root@nginx1 ~]#
[root@nginx1 ~]# /usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.6.2
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC)
TLS SNI support enabled
configure arguments: --with-debug --with-http_ssl_module --with-http_sub_module --with-http_stub_status_module --with-http_spdy_module
[root@nginx1 ~]#
[root@nginx1 ~]# stat /usr/local/nginx/sbin/nginx
File: `/usr/local/nginx/sbin/nginx'
Size: 5747928 Blocks: 11232 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 1053772 Links: 1
Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2015-02-06 09:34:24.078330934 +0200
Modify: 2015-02-06 09:34:11.094331557 +0200
Change: 2015-02-06 09:34:11.094331557 +0200
[root@nginx1 ~]#
Will attach new cacti graphs:
nginx-without-3rd-party.png
nginx-without-3rd-party-1.png
by , 10 years ago
Attachment: | nginx-without-3rd-party-1.png added |
---|
by , 10 years ago
Attachment: | nginx-without-3rd-party.png added |
---|
comment:6 by , 10 years ago
Hello,
I upgraded to version 1.7.9 and problem does not occur for several hours.
This version probably is not affected.
If the problem occurs a later, I will open new ticket.
Thank you for cooperation.
comment:7 by , 10 years ago
Hi guys,
I just feel free to leave an update here and to confirm that I am also experiencing this behavior with the latest mainline version 1.7.10. See latest attachment too see the increase of writing connections over a 7 day period. No negative side effects seen so far.
nginx version: nginx/1.7.10
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-mail --with-mail_ssl_module --with-file-aio --with-http_spdy_module --with-http_geoip_module --with-ipv6 --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic'
2.6.32-504.1.3.el6.x86_64 #1 SMP Tue Nov 11 17:57:25 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
by , 10 years ago
Attachment: | nginx-1.7.10-7-day.png added |
---|
7 day increase of writing connections with spdy
comment:8 by , 10 years ago
Could you provide your configuration? Do you have anything in the error log?
comment:9 by , 10 years ago
Hi Valentin,
phew, the set of configs is kind of massive. lots of individual configs in conf.d. error log is clean beside some upstream timeouts, no spdy errors at all. are you looking for anything specific?
comment:10 by , 9 years ago
I just want to report this issue with the latest stable version 1.8.0
nginx version: nginx/1.8.0
built by gcc 4.7.2 (Debian 4.7.2-5)
built with OpenSSL 1.0.1e 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-mail --with-mail_ssl_module --with-file-aio --with-http_spdy_module --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,--as-needed' --with-ipv6
$ uname -a
Linux cl-lb1 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u2 x86_64 GNU/Linux
We have turned spdy for our static content which makes about 70% of traffic. It was server by plain http before (and it is pretty amazing, that it works!). So we are not sure whether this is spdy issue or https. But we used to have some https traffic before (less then 1%) and writing connections were not leaking.
See the image:
The sudden drop is a restart. Just before switching spdy there was an outage of internet connection. So ignore the drop of waiting connections.
The main question is - before it gets fixed should we ignore the value leaking or do preventive restarts?
comment:11 by , 9 years ago
Reloading the configuration should be enough. But please note, that it's not something easy to find and fix, since it's not reproducible with simple configs or even in a test environment.
So I'm kindly asking again everybody who experiences this issue to provide as much information as possible: configuration, debug logs, etc. If someone can provide steps to reproduce, then I'll fix it very soon.
by , 9 years ago
Attachment: | nginx_status-day.png added |
---|
comment:12 by , 9 years ago
Restart is required, reload does not help visually. I was not sure, if it was https or spdy, so now I confirmed this to happen with spdy only. We have rather simple config files, may I mail them somewhere? On this chart it is perhaps interesting to see how it behaved when traffic was decreasing. It is also visible, that active connections are not following the waiting connections, when writing connections is leaking.
comment:14 by , 9 years ago
Keywords: | spdy added |
---|
comment:16 by , 9 years ago
Same problem here. Reloading configuration does not help, I have to restart nginx to fix the problem each time. My server crashed on this problem. SPDY is enabled. This is my config:
server { listen 443 default_server spdy; server_name -snip-; ssl on; ssl_certificate /etc/nginx/ssl.d/snip.chained.crt; ssl_certificate_key /etc/nginx/ssl.d/snip.key; ssl_stapling on; ssl_stapling_verify on; ssl_trusted_certificate /etc/nginx/ssl.d/ca.bundle.ocsp.pem; ssl_session_tickets on; ssl_session_timeout 5m; ssl_session_cache shared:SSL:5M; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS; ssl_prefer_server_ciphers on; ssl_dhparam /etc/nginx/ssl.d/dhparam.pem; root /snip/; index index.php index.html index.htm; open_file_cache max=2000 inactive=20s; open_file_cache_valid 60s; open_file_cache_min_uses 5; open_file_cache_errors off;
uname -a Linux snip 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Sep 15 15:05:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
nginx -V nginx version: nginx/1.6.3 built by gcc 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) TLS SNI support enabled configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-file-aio --with-ipv6 --with-http_ssl_module --with-http_spdy_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module --with-http_image_filter_module --with-http_geoip_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_stub_status_module --with-http_perl_module --with-mail --with-mail_ssl_module --with-pcre --with-pcre-jit --with-google_perftools_module --with-debug --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E'
yum info nginx Installed Packages Name : nginx Arch : x86_64 Epoch : 1 Version : 1.6.3 Release : 8.el7 Size : 1.4 M Repo : installed From repo : epel Summary : A high performance web server and reverse proxy server URL : http://nginx.org/ License : BSD Description : Nginx is a web server and a reverse proxy server for HTTP, SMTP, POP3 and : IMAP protocols, with a strong focus on high concurrency, performance and low : memory usage.
And my munin graph in the attachment.
I've got SELinux enabled on my server.
comment:17 by , 9 years ago
This looks like a duplicate of ticket #626, fixed in 1.9.12. Please try nginx 1.9.12 to see if it helps.
comment:18 by , 9 years ago
Yes it does seem like a duplicate of #626. I'm seeing these errors in my log when reloading:
2016/03/17 09:43:47 [alert] 11819#0: *43809 open socket #49 left in connection 4 2016/03/17 09:43:47 [alert] 11819#0: *43920 open socket #45 left in connection 9 2016/03/17 09:43:47 [alert] 11819#0: *49839 open socket #39 left in connection 17 2016/03/17 09:43:47 [alert] 11819#0: *44130 open socket #21 left in connection 29 2016/03/17 09:43:47 [alert] 11819#0: *39756 open socket #55 left in connection 34 2016/03/17 09:43:47 [alert] 11819#0: *41352 open socket #15 left in connection 44 2016/03/17 09:43:47 [alert] 11819#0: *40153 open socket #14 left in connection 58 2016/03/17 09:43:47 [alert] 11819#0: *44279 open socket #48 left in connection 64 2016/03/17 09:43:47 [alert] 11819#0: *69858 open socket #30 left in connection 73 2016/03/17 09:43:47 [alert] 11819#0: *40706 open socket #56 left in connection 77 2016/03/17 09:43:47 [alert] 11819#0: aborting
Shouldn't the EPEL version of Nginx be patched or something?
comment:19 by , 9 years ago
Resolution: | → duplicate |
---|---|
Status: | new → closed |
You can install nginx 1.9.12 from sources or using our packages, see http://nginx.org/en/download.html. Thanks for testing, closing this.
comment:20 by , 8 years ago
The bug has re-appeared since the last update of nginx. It was gone for some time, but reappeared in week 33/34.
Error log is filled with these:
2016/09/09 12:07:37 [alert] 22910#22910: *1682525 open socket #65 left in connection 121 2016/09/09 12:07:37 [alert] 22910#22910: *2481905 open socket #24 left in connection 142 2016/09/09 12:07:37 [alert] 22910#22910: *834242 open socket #43 left in connection 144 2016/09/09 12:07:37 [alert] 22910#22910: *985206 open socket #12 left in connection 163 2016/09/09 12:07:37 [alert] 22910#22910: *2529918 open socket #33 left in connection 165
nginx -V
nginx version: nginx/1.10.1 built by gcc 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) built with OpenSSL 1.0.1e-fips 11 Feb 2013 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_perl_module=dynamic --add-dynamic-module=njs-1c50334fbea6/nginx --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --with-http_v2_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic'
uname -a
Linux snip 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Sep 15 15:05:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
And my config nowadays:
server { listen 443 default_server ssl http2; server_name snip; ssl on; ssl_certificate snip.crt; ssl_certificate_key snip.key; ssl_stapling on; ssl_stapling_verify on; ssl_trusted_certificate snip.pem; ssl_session_tickets on; ssl_session_timeout 5m; ssl_session_cache shared:SSL:5M; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS; ssl_prefer_server_ciphers on; ssl_dhparam snip.pem; (...)
comment:21 by , 8 years ago
There is at least one connection leak in HTTP/2 as fixed in nginx 1.11.3, and the fix is not yet available in 1.10.x. Please try 1.11.x instead.
by , 6 years ago
Attachment: | nginx_leak.jpg added |
---|
comment:22 by , 6 years ago
Resolution: | duplicate |
---|---|
Status: | closed → reopened |
Have same issue with nginx 1.15.7
uname -a
Linux CentOS-72-64-minimal 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6 23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
nginx -V
nginx version: nginx/1.15.7 built by gcc 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) built with OpenSSL 1.0.2k-fips 26 Jan 2017 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-p ath=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path= /var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/v ar/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/v ar/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp -- http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var /cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user= nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_add ition_module --with-http_auth_request_module --with-http_dav_module --with-http_ flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_ mp4_module --with-http_random_index_module --with-http_realip_module --with-http _secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_ stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --wi th-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_m odule --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_ FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -W l,-z,now -pie'
comment:23 by , 6 years ago
Resolution: | → duplicate |
---|---|
Status: | reopened → closed |
Could you reproduce it without 3rd-party modules?