CLOSE_WAIT socket leak in downstream connections with keepalive

[Anton Ovchinnikov]

It looks like in certain situations Nginx doesn't close connections to downstream properly, and as a result, lots of sockets end up in CLOSE_WAIT state.

One of those situations: Nginx sitting behind an Envoy proxy (​https://www.envoyproxy.io/) and responding unconditionally (e.g. with return 200) or with activated rate limits (via limit_req), all that with keepalive enabled. Clients send POST requests with non-empty body to Envoy that proxies them to Nginx.

Here you can find a reproducible example with instructions: ​https://github.com/tonyo/nginx-too-many-close-wait/ Here's a basic Nginx configuration taken from there that was used to reproduce the issue:

worker_processes 1;

events {}

http {
  server {
    listen 8080;
    server_name _;

    return 200;
  }
}

I could reproduce the issue with all the version I tested with (1.14.2, 1.18.0, 1.19.7).

On the network level, it seems that the issue manifests itself when Nginx responds before fully reading the request body, and then doesn't properly react to TCP FIN packet received from the downstream.

I found this issue that seems related: ​https://forum.nginx.org/read.php?2,286665,286699#msg-286699 We briefly tested the provided patch and it does seem to work, but I don't really know how much we can rely on it, and whether it is still up-to-date.

Let me know if I can provide any more details.

[Maxim Dounin]

Thanks for the ticket. This is exactly the issue you've linked, and the patch by Sergey should still apply (a better link would be ​http://mailman.nginx.org/pipermail/nginx/2020-January/058867.html though). Thanks for your feedback on the patch as well.

Note though that there is no leak here: rather, a connection is kept open longer that it should given that TCP FIN is already received, and only closed once keepalive_timeout expires. This certainly needs to be fixed, but it's rather a suboptimal behaviour in a specific and not common use case, and hence low priority.

[Anton Ovchinnikov]

Thanks for your reply and explanation.

Do you perhaps also have insight into any potential dangers the issue might bring? In our setup in Google Cloud Platform keepalive_timeout is set to a pretty high value (620 seconds, as recommended by ​https://cloud.google.com/load-balancing/docs/https#timeouts_and_retries), so these CLOSE_WAIT connections might linger for more than 10 minutes. What we also observe is that they seem to be closed/recycled when the total number of connections (including the CLOSE_WAIT ones) for a given worker reaches its worker_connections limit. We haven't seen any hard crashes or 500s when this happens, but the question is, can we rely on Nginx here doing the "right" thing of closing these half-broken connections first, and not the active/live ones?

[Sergey Kandaurov <pluknet@…>]

In 7804:4a9d28f8f39e/nginx:

Cancel keepalive and lingering close on EOF better (ticket #2145).

Unlike in 75e908236701, which added the logic to ngx_http_finalize_request(),
this change moves it to a more generic routine ngx_http_finalize_connection()
to cover cases when a request is finalized with NGX_DONE.

In particular, this fixes unwanted connection transition into the keepalive
state after receiving EOF while discarding request body. With edge-triggered
event methods that means the connection will last for extra seconds as set in
the keepalive_timeout directive.

[Sergey Kandaurov]

Fix committed, thanks for report.

[Sergey Kandaurov]

Replying to Anton Ovchinnikov:

the question is, can we rely on Nginx here doing the "right" thing of closing these half-broken connections first, and not the active/live ones?

That's correct. When approaching the worker_connections limit, nginx is expected to close the least recently used keepalive connections.