Opened 7 years ago

Closed 6 years ago

Last modified 3 years ago

#346 closed defect (fixed)

Writing connection leak while SPDY enabled

Reported by: Marcus Maciel Owned by: Valentin V. Bartenev
Priority: minor Milestone:
Component: nginx-module Version: 1.3.x
Keywords: spdy Cc:
uname -a: Linux php01 3.2.0-37-virtual #58-Ubuntu SMP Thu Jan 24 15:48:03 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: root@php01:~# nginx -V
nginx version: nginx/1.4.0
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-mail --with-mail_ssl_module --with-file-aio --with-http_spdy_module --with-ipv6
root@php01:~#

Description

I'm trying to use SPDY module, but when I enable this into my config

listen 443 ssl spdy;

The number of Writing connections start to leak.

Attachments (3)

Screen Shot 2013-04-29 at 11.45.14 AM.png (182.1 KB ) - added by Marcus Maciel 7 years ago.
As you can see on attached file Writing connection still leaking until restart nginx without spdy option
nginx_status-week.png (24.9 KB ) - added by Marcus Maciel 7 years ago.
writting connections leak with nginx configured as a proxy when spdy is enabled.…
Screen Shot 2013-10-24 at 6.20.10 PM.png (133.8 KB ) - added by Marcus Maciel 6 years ago.
leak continues on 1.5.6

Download all attachments as: .zip

Change History (34)

by Marcus Maciel, 7 years ago

As you can see on attached file Writing connection still leaking until restart nginx without spdy option

comment:1 by Valentin V. Bartenev, 7 years ago

Owner: set to Valentin V. Bartenev
Status: newassigned

Could you provide the full config? Is there anything in the error log?

comment:2 by Marcus Maciel, 7 years ago

nginx.conf

root@php01:/etc/nginx# cat nginx.conf
user www-data;
worker_processes  auto;

error_log  /srv/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
	multi_accept on;
    	worker_connections 1024;
	use epoll;
}

http {
	include       /etc/nginx/mime.types;
	default_type  application/octet-stream;
	access_log	/srv/log/nginx/access.log;

        ssl_session_cache    shared:SSL:10m;
        ssl_session_timeout  10m;
        ssl_ciphers ALL:!kEDH!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;


       # Custom logging
        log_format custom '$remote_addr - $remote_user [$time_local]  '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent" CACHE:$upstream_cache_status RESPONSE:$upstream_response_time';

	
	fastcgi_cache_path /srv/cache/nginx levels=1:2 keys_zone=underlinux:512m inactive=30m max_size=1024m;
	fastcgi_buffer_size   8192k;
        fastcgi_buffers   4 8192k;
        fastcgi_busy_buffers_size   8192k;
	sendfile       on;
	tcp_nopush     on;
	tcp_nodelay off;
	client_body_buffer_size 8192k;
	client_max_body_size 500m;
	fastcgi_send_timeout 300;
	fastcgi_read_timeout 300;
	keepalive_timeout 15 15;
	keepalive_requests 100;
	ignore_invalid_headers on;

	gzip  on;
	gzip_comp_level 9;
	gzip_types		text/plain text/css text/javascript text/xml application/x-javascript application/xml application/xml+rss;
	gzip_buffers   64 8k;
	gzip_http_version 1.0;
	gzip_min_length  1100;
	gzip_proxied any;
	gzip_vary on;
	server_tokens off;

#	include /etc/nginx/conf.d/*.conf;
	include /etc/nginx/sites-enabled/*;
}

root@php01:/etc/nginx# 


root@php01:/etc/nginx/sites-enabled# cat under-linux.org.conf 
server {
	listen 80;
	access_log   off;
	server_name *.under-linux.org underlinux.com.br *.underlinux.com.br under-linux.net *.under-linux.net underlinux.net *.underlinux.net underlinux.org *.underlinux.org under-linux.org;

	rewrite /forums/(.*)/(\d+)-(.*).html https://under-linux.org/showthread.php?t=$2 permanent;
        rewrite /noticias/(.*)-(\d+) https://under-linux.org/content.php?r=$2 permanent;
        rewrite /forums/(.*)-(\d+) https://under-linux.org/showthread.php?t=$2 permanent;
        rewrite /blogs/(.*)/(.*)-(\d+) https://under-linux.org/entry.php?b=$3 permanent;
        rewrite /f(\d+)/(.*)-(\d+)/(.*) https://under-linux.org/showthread.php?t=$3 permanent;
        rewrite /f(\d+)/(.*)-(\d+) https://under-linux.org/showthread.php?t=$3 permanent;
        rewrite /f(\d+)-(.*)/(.*) https://under-linux.org/showthread.php?t=$1 permanent;
        rewrite /f(\d+)-(.*) https://under-linux.org/showthread.php?t=$1 permanent;
        rewrite /content/(.*)-(\d+) https://under-linux.org/content.php?r=$2 permanent;
        rewrite /b(\d+)-(.*) https://under-linux.org/entry.php?b=$1 permanent;
	rewrite /(.*)-(\d+)/(.*)-(\d+) https://under-linux.org/showthread.php?t=$4 permanent;
	rewrite /(.*)-(\d+) https://under-linux.org/content.php?r=$2 permanent;
	rewrite /forums/(\d+)-(.*) https://under-linux.org/showthread.php?t=$1 permanent;
	rewrite ^	https://under-linux.org$request_uri? permanent;
}
server {
	index index.php index.html;
	server_name under-linux.org;
	#ssl		on;
    	listen              443 ssl;
	ssl_certificate     under-linux.org.chained.crt;
	ssl_certificate_key underlinux.key;

	access_log /srv/log/nginx/underlinux-nginx-access.log custom;
	error_log /srv/log/nginx/underlinux-nginx-error.log notice;
	root /srv/www/01/under-linux.org/public_html;

	rewrite /noticias/(.*)-(\d+) /content.php?r=$2 permanent;
	rewrite /forums/(.*)-(\d+) /showthread.php?t=$2 permanent;
	rewrite	/blogs/(.*)/(.*)-(\d+) /entry.php?b=$3 permanent;
	rewrite /f(\d+)/(.*)-(\d+)/(.*) /showthread.php?t=$3 permanent;
	rewrite /f(\d+)/(.*)-(\d+) /showthread.php?t=$3 permanent;
	rewrite /f(\d+)-(.*)/(.*) /showthread.php?t=$1 permanent;	
	rewrite /f(\d+)-(.*) /showthread.php?t=$1 permanent;
	rewrite /content/(.*)-(\d+) /content.php?r=$2 permanent;
	rewrite /b(\d+)-(.*) /entry.php?b=$1 permanent;

        location = /robots.txt {
                expires       -24h;
        }


    location ~ /paineladmin/ {
       		auth_basic "Hello, Please login";
       		auth_basic_user_file /etc/nginx/underpass;
        location ~ \.php$ {
                fastcgi_pass  127.0.0.1:9000;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include fastcgi_params;
        }


    }



    location ~ /forumrunner/ {
        location ~ \.php$ {
                fastcgi_pass  127.0.0.1:9000;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include fastcgi_params;
        }
    }

rewrite ^/((urllist|sitemap).*\.(xml|txt)(\.gz)?)$ /vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 last;

#Inicio set nocache var
		set $nocache "";
		if ($http_cookie ~ (bb_userid|bb_imloggedin|vbseo_loggedin)) {
		  set $nocache "Y";
		}

             if ($request_uri ~ "(.*)(cron\.php|ajax\.php|search\.php|image\.php|login\.php|register\.php|usercp\.php|private\.php|sendmessage\.php|faleconosco|vbseo-xmlrpc|profile\.php|forumrunner\/|paineladmin\/)(.*)") {
		set $nocache "Y";                
		}

#
             if ($request_uri ~ "(.*)(attachment\.php|attachments\/)(.*)") {
                set $nocache "";
                }
#end of nocache

	# We don't want to allow the browsers to see .hidden linux/unix files
	location ~ /\. { deny  all; access_log off; log_not_found off; }

	location ~* \.(js|bmp|pdf|swf|css|jpg|jpeg|gif|png|ico)$ {
		access_log   off;
                add_header Pragma public;
                add_header Cache-Control "public, must-revalidate, proxy-revalidate";
		expires      max;
	}

        location = /activity.php {
                fastcgi_pass  127.0.0.1:9000;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include fastcgi_params;
        }

        location = /thanks.php {
                fastcgi_pass  127.0.0.1:9000;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include fastcgi_params;
        }

	location = /attachment.php {
		access_log   off;
                fastcgi_pass  127.0.0.1:9000;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include fastcgi_params;
                add_header Pragma public;
                add_header Cache-Control "public, must-revalidate, proxy-revalidate";
                expires      max;
                #novas configs de cache
                fastcgi_ignore_headers Cache-Control Expires Set-Cookie;
                fastcgi_hide_header Set-Cookie;
                fastcgi_cache_valid   200 302  5m;
                fastcgi_cache_valid   301      5m;
                fastcgi_cache_valid   any      1m;
                fastcgi_cache_min_uses  1;
                fastcgi_cache_use_stale error  timeout invalid_header http_500 http_503;
                fastcgi_cache underlinux;
                fastcgi_cache_key $request_method$request_uri$cookie_bb_userid$cookie_bb_imloggedin;
                #end of cache part

        }



location / {

	error_page     404 = @fixcms;

        location ~ \.php$ {
         	if ($nocache !~ "Y") {
                	return 404;
           	}
        	error_page     404 = @cache;
		fastcgi_pass  127.0.0.1:9000;
		fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include fastcgi_params;
        }

}
	location @fixcms {
		rewrite /(.*)-(\d+) /content.php?r=$2 permanent;	
	}

        location @cache {
                #novas configs de cache
                fastcgi_ignore_headers Cache-Control Expires Set-Cookie;
		fastcgi_hide_header Set-Cookie;
                fastcgi_no_cache $nocache;
                fastcgi_cache_bypass $nocache;
                fastcgi_cache_valid   200 302  5m;
                fastcgi_cache_valid   301      5m;
                fastcgi_cache_valid   any      1m;
                fastcgi_cache_min_uses  1;
                fastcgi_cache_use_stale error  timeout invalid_header http_500 http_503;
                fastcgi_cache underlinux;
                fastcgi_cache_key $request_method$request_uri$cookie_bb_userid$cookie_bb_imloggedin;
                #end of cache part
                fastcgi_pass  127.0.0.1:9000;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include fastcgi_params;
        }

        location @fastcache {
                #novas configs de cache
                fastcgi_ignore_headers Cache-Control Expires Set-Cookie;
                fastcgi_hide_header Set-Cookie;
                fastcgi_no_cache $nocache;
                fastcgi_cache_bypass $nocache;
                fastcgi_cache_valid   200 302  5m;
                fastcgi_cache_valid   301      5m;
                fastcgi_cache_valid   any      1m;
                fastcgi_cache_min_uses  1;
                fastcgi_cache_use_stale error  timeout invalid_header http_500 http_503;
                fastcgi_cache underlinux;
                fastcgi_cache_key $request_method$request_uri$cookie_bb_userid$cookie_bb_imloggedin;
                #end of cache part
                fastcgi_pass  127.0.0.1:9000;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include fastcgi_params;
        }

}
Last edited 6 years ago by Valentin V. Bartenev (previous) (diff)

comment:3 by Marcus Maciel, 7 years ago

Hello Valentin,

Let me know if you need any other information to help me on this bug.

Thanks,

comment:4 by Marcus Maciel, 7 years ago

any update ???

comment:5 by Christopher O'Connell, 7 years ago

I can confirm this happening in 1.4.0 with a substantially similar configuration to that already posted. Please let me know if there's any help I can provide in testing with current code or proposed patches.

comment:6 by Valentin V. Bartenev, 7 years ago

Priority: criticalminor

Investigation of the issue will take some time. It seems that the problem only appears on certain configurations under some conditions. There is also a known incompatibility with spdy and caching enabled, but it should not cause such leaks.

in reply to:  6 comment:7 by Christopher O'Connell, 7 years ago

Replying to Valentin V. Bartenev:

Investigation of the issue will take some time. It seems that the problem only appears on certain configurations under some conditions. There is also a known incompatibility with spdy and caching enabled, but it should not cause such leaks.

Can you provide any guidance on what configuration commands or values produce the leaks?

comment:8 by Danny Lockard, 7 years ago

I am also seeing a steady upward trend of open "writing" connections in my nginx status. We have a similar, less complex configuration.

server {
  listen IP:80;
  rewrite ^(.*) https://site$request_uri? permanent;
}

server {
  listen IP:443 ssl spdy;

    if ($host ~* www\.(.*)) {
      rewrite ^(.*) https://site$request_uri? permanent;
    }

  ssl_certificate /etc/nginx/ssl/site.crt ;
  ssl_certificate_key /etc/nginx/ssl/site.key;

  client_max_body_size 64M;

  access_log /var/log/nginx/site.access.log;

  location /nginx_status {
    stub_status on;
    access_log   off;
    allow 127.0.0.1;
    deny all;
  }

  location / {
    proxy_pass http://site-backend;
    proxy_set_header X-Real-IP  $remote_addr;
    proxy_set_header Host $host;
    location ~* \.(png|jpg|jpeg|gif|ico|bmp|swf)$ {
      expires 48h;
      log_not_found off;
      proxy_pass http://site-backend;    
      proxy_set_header X-Real-IP  $remote_addr;
      proxy_set_header Host $host;
      add_header Cache-Control public;
    }

    location ~* \.(js|css)$ {
      expires 24h;
      log_not_found off;
      proxy_pass http://site-backend;    
      proxy_set_header X-Real-IP  $remote_addr;
      proxy_set_header Host $host;
      add_header Cache-Control public;
    }

    location ~* \.(html|htm|xml)$ {
      expires 1h;
      log_not_found off;
      proxy_pass http://site-backend;
      proxy_set_header X-Real-IP  $remote_addr;
      proxy_set_header Host $host;
    }

    location /assets/files/ {
      expires 4000h;
      log_not_found off;
      proxy_pass http://site-backend;
      proxy_set_header X-Real-IP  $remote_addr;
      proxy_set_header Host $host;
      add_header Cache-Control public;
    }

    location /assets/content/ {
      expires 4000h;
      log_not_found off;
      proxy_pass http://site-backend;
      proxy_set_header X-Real-IP  $remote_addr;
      proxy_set_header Host $host;
      add_header Cache-Control public;
    }

    location /assets/ {
      expires 48h;
      log_not_found off;
      proxy_pass http://site-backend;
      proxy_set_header X-Real-IP  $remote_addr;
      proxy_set_header Host $host;
    }
  }

  location /externalmedia {
    proxy_pass http://static-storage:80;
    proxy_set_header X-Real-IP  $remote_addr;
    proxy_set_header Host $host;
  }
}

comment:9 by Marcus Maciel, 7 years ago

Just using nginx as proxy enabling spdy I have same issue so I think this is a Major issue and not a minor issue like was reclassified .... Since in my understand when nginx reach limit of writting connections will stop to accept new connections right ?

by Marcus Maciel, 7 years ago

Attachment: nginx_status-week.png added

writting connections leak with nginx configured as a proxy when spdy is enabled....

comment:10 by Valentin V. Bartenev, 7 years ago

Am I right that you all have nothing in your error logs?

Replying to Marcus Maciel <mmaciel@gmail.com>:

Since in my understand when nginx reach limit of writting connections will stop to accept new connections right ?

Not quite exactly. It just a counter in the stub status module that can be in a wrong state, and in case of spdy it does not correspond with real connections.

comment:11 by Marcus Maciel, 7 years ago

I could not find anything on error logs... this should happen in any installation with spdy apparently.

comment:12 by Christopher O'Connell, 7 years ago

I examined 2 machines exhibiting the problem in the status module and neither showed any unexpected errors in the error logs.

Thanks for clarifying that this is just an invalid counter and not a bug in the SSL implementation.

comment:13 by Valentin V. Bartenev, 7 years ago

I actually just explained that the growth of "writing" counter may not mean literally leaking of connections. But since I still cannot reproduce the issue even using similar configs I can only guess what's wrong.

If somebody would be able to reduce its case to something simple and reproducible form scratch, then it will help a lot.

comment:14 by Marcus Maciel, 7 years ago

Valentin, it actually take sometime to start the leak and in a lab environment without too many connections you probably will never see this issue.

You will need some good traffic running for some time to start to see this leak...

comment:15 by Edward Ackroyd, 6 years ago

I can also reproduce this in 1.4.2: "Active connections" and "Writing" too high. This server has little traffic (used to have lots):

Active connections: 1110 
server accepts handled requests
 220221922 220221922 169671639 
Reading: 0 Writing: 1759 Waiting: 5


However, there are by no means that many "Active connections":

# netstat -n | egrep "tcp.*:(443|80) " | grep -o "[A-Z_]*" | sort | uniq -c
    222 ESTABLISHED
      8 TIME_WAIT

Eventually "Active connections" reaches worker_connections*worker_processes max and no more "Active connections" available.

Steps to reproduce: Set up nginx with SPDY, make lots of requests (from lots of different clients) (stop the traffic and see that Active connections and Writing never go down).

Here is my nginx.conf:

pid	nginx.pid;

worker_processes  6;

events {
    worker_connections  1024;
}

http {	
	keepalive_timeout 10s;
                    
	upstream up {
		server localhost:8080;
		keepalive 150;
	}
	
	server {
		listen 443 ssl spdy default_server;
		ssl_certificate      X.crt;
		ssl_certificate_key  X.pem;

		charset utf-8;

		location / {
			proxy_pass			http://up;
			proxy_http_version	1.1;
			proxy_set_header	Connection "";
			proxy_redirect		http:// https://;
			proxy_set_header	Host $host;
			proxy_set_header	X-IP $remote_addr;
		}
		
		location /nginx_status {
			stub_status on;
			access_log   off;
		}
	}
	
	
	server {
		listen 80;

		charset utf-8;

		location / {
			proxy_pass			http://up;
			proxy_http_version	1.1;
			proxy_set_header	Connection "";
			proxy_set_header	Host $host;
			proxy_set_header	X-IP $remote_addr;
		}
	}
}
Last edited 6 years ago by Edward Ackroyd (previous) (diff)

comment:16 by Valentin V. Bartenev, 6 years ago

It seems I found one case that can lead to such leaks. Could you test the patch below?

  • src/http/ngx_http_spdy.c

    diff -r 50f065641b4c src/http/ngx_http_spdy.c
    a b ngx_http_spdy_create_stream(ngx_http_spd  
    18141814
    18151815    rev->data = fc;
    18161816    rev->ready = 1;
    1817     rev->handler = ngx_http_empty_handler;
     1817    rev->handler = ngx_http_test_reading;
    18181818    rev->log = log;
    18191819
    18201820    ngx_memcpy(wev, rev, sizeof(ngx_event_t));
    18211821
    18221822    wev->write = 1;
     1823    wev->handler = ngx_http_empty_handler;
    18231824
    18241825    ngx_memcpy(fc, sc->connection, sizeof(ngx_connection_t));
    18251826

It is for 1.4.2, but for the other versions it should be applied well with a little offset. So anyone who has encounter this problem feel free to try the fix.

comment:17 by Edward Ackroyd, 6 years ago

The patch does not compile:

src/http/ngx_http_spdy.c: In function ‘ngx_http_spdy_create_stream’:
src/http/ngx_http_spdy.c:1817:18: error: assignment from incompatible pointer type [-Werror]

changed code block after patch:

    rev->data = fc;
    rev->ready = 1;
    rev->handler = ngx_http_test_reading;
    rev->log = log;

    ngx_memcpy(wev, rev, sizeof(ngx_event_t));

    wev->write = 1;
    wev->handler = ngx_http_empty_handler;
Last edited 6 years ago by Edward Ackroyd (previous) (diff)

in reply to:  17 comment:18 by Valentin V. Bartenev, 6 years ago

Oops, sorry, it's wrong patch indeed. Accidentally I've built nginx without the changes and then believed that it works.

Here is a new patch:

  • src/http/ngx_http_spdy.c

    diff -r 50f065641b4c src/http/ngx_http_spdy.c
    a b static ngx_int_t ngx_http_spdy_construct  
    145145static void ngx_http_spdy_run_request(ngx_http_request_t *r);
    146146static ngx_int_t ngx_http_spdy_init_request_body(ngx_http_request_t *r);
    147147
     148static void ngx_http_spdy_close_stream_handler(ngx_event_t *ev);
     149
    148150static void ngx_http_spdy_handle_connection_handler(ngx_event_t *rev);
    149151static void ngx_http_spdy_keepalive_handler(ngx_event_t *rev);
    150152static void ngx_http_spdy_finalize_connection(ngx_http_spdy_connection_t *sc,
    ngx_http_spdy_create_stream(ngx_http_spd  
    18141816
    18151817    rev->data = fc;
    18161818    rev->ready = 1;
    1817     rev->handler = ngx_http_empty_handler;
     1819    rev->handler = ngx_http_spdy_close_stream_handler;
    18181820    rev->log = log;
    18191821
    18201822    ngx_memcpy(wev, rev, sizeof(ngx_event_t));
    ngx_http_spdy_create_stream(ngx_http_spd  
    18791881}
    18801882
    18811883
     1884static void
     1885ngx_http_spdy_close_stream_handler(ngx_event_t *ev)
     1886{
     1887    ngx_connection_t    *fc;
     1888    ngx_http_request_t  *r;
     1889
     1890    fc = ev->data;
     1891    r = fc->data;
     1892
     1893    ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
     1894                   "spdy close stream handler");
     1895
     1896    ngx_http_spdy_close_stream(r->spdy_stream, 0);
     1897}
     1898
     1899
    18821900static ngx_http_spdy_stream_t *
    18831901ngx_http_spdy_get_stream_by_id(ngx_http_spdy_connection_t *sc,
    18841902    ngx_uint_t sid)
    ngx_http_spdy_finalize_connection(ngx_ht  
    28052823
    28062824    c->error = 1;
    28072825    c->read->handler = ngx_http_empty_handler;
     2826    c->write->handler = ngx_http_empty_handler;
    28082827
    28092828    sc->last_out = NULL;
    28102829

comment:19 by Edward Ackroyd, 6 years ago

Thanks Valentin. I have a server running the patched version, initial values look good. Will report back once a good amount of traffic has past through the server.

comment:20 by Edward Ackroyd, 6 years ago

The patch does not fix the issue. Here is a dump of nginx_status and sockets used by nginx:

Active connections: 437
server accepts handled requests
 2504744 2504744 2418189
Reading: 0 Writing: 563 Waiting: 148

    203 CLOSE_WAIT
      2 CLOSING
    158 ESTABLISHED
      4 FIN_WAIT
     79 TIME_WAIT

"Active connections" is wrong, even if you count those in CLOSE_WAIT
"Writing" is wrong, can't be larger than Active connections
In fact I think the only correct value is "requests".

In addition, when nginx is configured to use SPDY and proxy_cache it leaks (does not close) sockets in a rate of about 100 for every million requests (see the 203 in CLOSE_WAIT above). Does not do this when configured without proxy_cache.

in reply to:  20 comment:21 by Valentin V. Bartenev, 6 years ago

Replying to Edward Ackroyd <edward.ackroyd@gmail.com>:

"Writing" is wrong, can't be larger than Active connections

It looks like you misunderstand this counter. The "Writing" and "Reading" counters are about requests, not about connections. With SPDY you can have a lot of requests with "Writing" state in one connection, and this is exactly what SPDY is for.

So with SPDY Writing+Reading request counters can be several times larger than Active connections.

comment:22 by Edward Ackroyd, 6 years ago

For "Writing", in what time frame does nginx count the requests? What does a value of 563 really mean?

Do you agree the value for "Active connections" is wrong?

in reply to:  22 comment:23 by Valentin V. Bartenev, 6 years ago

Replying to Edward Ackroyd <edward.ackroyd@gmail.com>:

For "Writing", in what time frame does nginx count the requests? What does a value of 563 really mean?

Every request from start reading request headers till the end of reading request headers is counted as "Reading". All the rest of the processing of the request (including processing of upstream and reading request body) is counted in "Writing".

This may seem strange, but there are historical reasons for this, and it is one of the reasons why the module is called "stub status", and why it is not documented in the official documentation.

Do you agree the value for "Active connections" is wrong?

Yes, it looks like the problem still isn't fixed.

comment:24 by Valentin V. Bartenev, 6 years ago

http://mailman.nginx.org/pipermail/nginx-devel/2013-September/004204.html is relevant to the ticket.

A patch that can fix the problem:

  • src/http/ngx_http_spdy.c

    diff -r 50f065641b4c src/http/ngx_http_spdy.c
    a b ngx_http_spdy_state_data(ngx_http_spdy_c  
    12041204        }
    12051205
    12061206        if (rb->post_handler) {
     1207            r->read_event_handler = ngx_http_block_reading;
    12071208            rb->post_handler(r);
    12081209        }
    12091210    }
    ngx_http_spdy_read_request_body(ngx_http  
    26042605
    26052606    r->request_body->post_handler = post_handler;
    26062607
     2608    r->read_event_handler = ngx_http_test_reading;
     2609    r->write_event_handler = ngx_http_request_empty_handler;
     2610
    26072611    return NGX_AGAIN;
    26082612}
    26092613

comment:25 by Valentin V. Bartenev, 6 years ago

Resolution: fixed
Status: assignedclosed

The issue should be fixed by revision 4d0c70541784 (released in nginx 1.5.6).

comment:26 by Marcus Maciel, 6 years ago

In my understanding this is not fixed.

I'm running 1.5.6 and I still have this problem.

by Marcus Maciel, 6 years ago

leak continues on 1.5.6

comment:27 by Marcus Maciel, 6 years ago

I'm using the package from

deb http://nginx.org/packages/mainline/ubuntu/ precise nginx
deb-src http://nginx.org/packages/mainline/ubuntu/ precise nginx

comment:28 by Marcus Maciel, 6 years ago

Resolution: fixed
Status: closedreopened

in reply to:  26 comment:29 by Valentin V. Bartenev, 6 years ago

Replying to Marcus Maciel <marcus@underlinux.com.br>:

In my understanding this is not fixed.

I'm running 1.5.6 and I still have this problem.

Could you provide debug log with leaking connections in 1.5.6?

P.S. If you use packages from our repository, then you can install nginx-debug.

Last edited 6 years ago by Valentin V. Bartenev (previous) (diff)

comment:30 by Valentin V. Bartenev, 6 years ago

Resolution: fixed
Status: reopenedclosed

A writing connection leak was fixed in a279d2a33dbf (will be released with 1.5.8).

comment:31 by Maxim Dounin, 3 years ago

sensitive: 10
Note: See TracTickets for help on using tickets.