mirror + limit_req = writing connections

[urusha@…]

Hello,
Nginx seems to have a bug with mirror+limit_req
Configuration:

All servers could be the same for testing purposes (127.0.0.1)
Frontend server

limit_req_zone $binary_remote_addr zone=one:10m rate=5r/s;
location = /url1
{
  mirror /url2;
  proxy_pass http://127.0.0.1/test;
}
location = /url2
{
  internal;
  limit_req zone=one burst=10;
  proxy_pass http://127.0.0.1/test2;
}
location = /status { stub_status on; }

Backend server

location = /test { return 200; }

Mirror server

location = /test2 { return 200; }

Now run:

# for i in {1..1000}; do curl http://127.0.0.1/url1 >/dev/null & sleep 0.05; done

Wait for completion of all requests and see writing connections:

# curl http://127.0.0.1/status
Active connections: 271 
server accepts handled requests
 2001 2001 2001 
Reading: 0 Writing: 271 Waiting: 0
# sleep 120
# netstat -atn | grep 127.0.0.1:80 | grep -v CLOSE_WAIT | wc -l
270
# service nginx reload
# pgrep -f shutting
# netstat -atn | grep 127.0.0.1:80 | grep -v CLOSE_WAIT | wc -l
0
# curl http://127.0.0.1/status
Active connections: 271 
server accepts handled requests
 2002 2002 2002 
Reading: 0 Writing: 271 Waiting: 0 

When /url1 doesn't have limit_req, but /url2 has, number of writing connections from stub status begins to grow. Watching netstat, I can also see CLOSE_WAIT connections growing. I did't find any impact on requests processing at least when the number of connections is quite low. Actually, after reloading nginx there seems to be no real connections (writing). But this breaks nginx monitoring. Restart of nginx only resets writing connections number.
If both /url1 and /url2 have limit_req, or /url1 only has limit_req - all is OK.

We use amd64 debian stretch, with the nginx-extras package from debian buster (rebuilt on stretch).

[urusha@…]

Sorry, I mean

netstat -atn | grep 127.0.0.1:80 | grep CLOSE_WAIT | wc -l

instead of

netstat -atn | grep 127.0.0.1:80 | grep -v CLOSE_WAIT | wc -l

[Maxim Dounin]

Thanks for reporting this, I was able to reproduce this. There is a socket leak in the provided configuration because the timer as set by the limit_req in the mirror subrequest is executed in the context of the active request, and it does not wake up mirror subrequest. This needs to be addressed.