stub_status counter leak when killing old workers

[mookie-@…]

Hello,

we've started killing (sending SIGTERM) "old" nginx workers (nginx: worker process is shutting down) as we have regular configuration changes and a lot of websocket connections.

Since we do this, the counters from stub_status are incorrect.

server status

> curl localhost/nginx_status
Active connections: 65369
server accepts handled requests
 1042173178 1042173178 5035465167
Reading: 0 Writing: 31968 Waiting: 5356

Adding up Writing and Waiting it's just 37324. But even this count is too high. The correct number should be around this:

> ss | grep https | wc -l
7327

It's reproducable e.g. using echo.websocket.org:

nginx configuration

map $http_upgrade $connection_upgrade {
	default upgrade;
	''      close;
}

server {
	listen 1234;
	location / {
		proxy_set_header  Host echo.websocket.org;
		proxy_ssl_server_name on;
		proxy_ssl_name echo.websocket.org;
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection $connection_upgrade;

		proxy_pass https://echo.websocket.org:443;
	}
	location = /basic_status {
		    stub_status;
	}
}

Test

1. Open 2 tabs in the browser with the URL: http://localhost:1234/.ws
2. Now there are 3 active connection (one connection is the request to /basic_status):

# curl localhost:1234/basic_status
Active connections: 3
server accepts handled requests
 4 4 4
Reading: 0 Writing: 3 Waiting: 0

3. After that, reload the nginx process and you can see a nginx: worker process is shutting down process:

# systemctl reload nginx

# ps aux | grep [n]ginx
root     2233850  0.0  0.0  55372  5652 ?        Ss   22:29   0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 2233851  0.0  0.0  55896  6044 ?        S    22:29   0:00 nginx: worker process is shutting down
www-data 2233940  0.0  0.0  55868  5400 ?        S    22:30   0:00 nginx: worker process
www-data 2233941  0.0  0.0  55868  5240 ?        S    22:30   0:00 nginx: worker process

# kill 2233851

4. After we've killed the old process the websocket client in the browser will reconnect. After that we get two additional connection from stub_status even if the old connections are gone:

# curl localhost:1234/basic_status
Active connections: 5
server accepts handled requests
 7 7 7
Reading: 0 Writing: 5 Waiting: 0

[mookie-@…]

I've tested it with the latest version and migrated this to github: ​https://github.com/nginx/nginx/issues/131

Please close this ticket.