proxy_next_upstream_tries might be ignored with upstream keepalive

[fischerthiebaut@…]

there is a bug with proxy_next_upstream_tries, which is ignored if Nginx is under load and the upstream server close the connection prematurely.
It seems to only occurs with a connection closed.
If too many requests are producing this error, It is possible to bring all the upstreams down for a certain time.

Here is a reproducer on docker, but the issue was noticed on debian9/11 as well with nginx 1.10<=1.23.0

​https://github.com/tibz7/nginx_next_upstream_retries_bug

[Maxim Dounin]

The ​proxy_next_upstream_tries directive limits the number of upstream servers tried by nginx. In your case, as per the no live upstreams error as seen in the logs, all upstream servers are considered unavailable, and therefore there are no servers to try. If you want nginx to ignore errors and don't consider servers to be unavailable after errors, consider tuning ​max_fails and ​fail_timeout.

[fischerthiebaut@…]

Yes, max_fails AND fail_timeout are used in my configuration. I give more details about the bug in the repository.
Of course it is possible that all upstreams are down, that is a expectable consequence, not the bug itself.
The real issue is that under load these configs are by passed.
therefore instead of seeing 2 errors messages (as set by proxy_next_upstream_tries)
there will be one per upstream (so 6 in my example.)
Which can then naturally lead to an no live upstream error.

[Maxim Dounin]

Sorry, I've misread what you are actually complaining about.

It looks like your configuration uses keepalive connections to upstream servers, and the error you are generating is the one non-distinguishable from ​asynchronous close events. As such, when a cached connection to an upstream server is used, nginx retries request regardless of the proxy_next_upstream_tries till all servers are tried (or till a non-cached connection is used if there is only one upstream server). See ticket #2022 for details.

In recent HTTP standard editions requirements for retrying requests are ​relaxed compared to RFC 2616 referenced above, and it might make sense to relax nginx behaviour somehow, so it won't unconditionally retry such requests.

[fischerthiebaut@…]

Thank you for your reply. So if I under correctly this is not a bug, it is a feature?
I'm still dont completely understand why this behavior would happen only under load ?
It will try the first cached connection for the first available upstream, then once its close why does it need to try all of them and by pass proxy_next_upstream_tries?

[Maxim Dounin]

RFC 2616 requires clients to retry requests if a requests used a cached connection, and it was closed by the server when the client was sending the request.

To make sure such requests are retried, nginx retries them unconditionally, ignoring the number of tries left (which is normally a number of servers in the upstream, but can be overridden by set to a lower value with proxy_next_upstream_tries). Most notably, this makes it possible to retry requests to an upstream with just one server (normally, such requests are never retried). As a side effect, though, this makes proxy_next_upstream_tries ineffective for such requests (since the number of tries is ignored, see above).

That is, as implemented, nginx will work as follows:

• It will select a server to use per request balancing method configured.
• If there are cached connections to this server, it'll use one.
• If a cached connection is closed while nginx was sending the request, nginx will retry the request, repeating the above steps.

For upstreams with just one server, this essentially means that nginx will retry till there are cached connections. For upstreams with multiple servers, nginx will keep track of servers it tried, and the first step will fail as long as all servers are tried.

Hope this better explains how it works now, and why the side effects observed do appear. It's an open question how to change this behaviour to make it less surprising.

[fischerthiebaut@…]

ok thank you for explanation. But for our point of view, it is still suprising that even with max set we can bring all upstream down.

[Konrad Baumgart]

Retries after asynchronous close events need a limit.

I found a request with 9707 attempts to connect to upstream in my log, the truncated json log looks like:

{
  "upstream_status": "502, 502, 502, <…>, 502, -",
  "upstream_response_time": "0.003, 0.001, 0.002, <…>, 0.002, 0.002",
  "request_time": 14.561,
  "status": 499,
  <…>

and errors look like

2024/06/07 11:47:25 [error] 52738#52738: *48268611 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 37.….….…, server: …, request: "GET …

I have a busy nginx server 1.21.6 (inside ingress-nginx 1.9.4), proxy_next_upstream_tries is set to 2 and the poisoned request caused upstream to break connections while reading the request line