Opened 3 years ago

Closed 3 years ago

#2192 closed defect (duplicate)

Upstream Keepalive - TCP connection to the upstream closes

Reported by: https://stackoverflow.com/users/10634976/patrick-scott-best Owned by:
Priority: minor Milestone:
Component: nginx-core Version: 1.19.x
Keywords: upstream keepalive Cc:
uname -a: Linux srv001b-media-01 5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.20.0
built by gcc 8.3.0 (Debian 8.3.0-6)
built with OpenSSL 1.1.1d 10 Sep 2019
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-1.20.0/debian/debuild-base/nginx-1.20.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

Description

I have an NGINX upstream definition that involves rotation through several servers to find content. The content requested is only on one of these servers, so I keep "trying" until I find the content, or die after I've exhausted all upstreams. This works well, however, there is a delay in "fishing through" any upstream servers that don't have the content to find the correct one due to TCP overhead.

The server on which the content is found has successfully established a connection using the keepalive / HTTP 1.1 method.

The problem I have is that NGINX closes the TCP connection to any upstream server that returns a 404. Thus, the next time around, a TCP connection must be established before retrieving another 404 and moving on to (the correct) webserver.

When a connection is established with the correct upstream server that bears 200-fruit, that upstream connection is successfully established with keepalive, and subsequent attempts at the content are performed through this established connection as expected. This is good - packet capture and netstats show the longevity of this connection for multiple subsequent HTTP requests.

My problem is that I would like to more rapidly run through the list of (bad) servers, and not have to wait for another TCP connection to be brought up each and every "try".

The Downstream NGINX server is the one that initiates the closure of the TCP connection with FIN ACK, not the upstream.

Downstream Config:

proxy_socket_keepalive on;

upstream myupstream {

server 10.x.x.x:32700 max_fails=0;
server 10.y.y.y:32700 max_fails=0;
server 10.z.z.z:32700 max_fails=0;
keepalive 32;

}

location myfile.jpg {

proxy_set_header Connection "";
proxy_http_version 1.1;
proxy_intercept_errors on;
proxy_pass http://myupstream;
proxy_next_upstream error timeout http_500 http_502 http_503 http_404;
proxy_next_upstream_tries 3;
proxy_read_timeout 2s;

}

Even the upstream server desires to keep this connection open, but NGINX downstream terminates the TCP connection regardless.

HTTP/1.1 404 Not Found
Server: nginx
Date: Thu, 20 May 2021 22:44:20 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive

<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

Attachments (1)

Upstream Keepalive.png (29.1 KB ) - added by https://stackoverflow.com/users/10634976/patrick-scott-best 3 years ago.
Wireshark clip that demonstrates the downstream is terminating the TCP connection.

Download all attachments as: .zip

Change History (3)

by https://stackoverflow.com/users/10634976/patrick-scott-best, 3 years ago

Attachment: Upstream Keepalive.png added

Wireshark clip that demonstrates the downstream is terminating the TCP connection.

comment:1 by Maxim Dounin, 3 years ago

Similarly to error interception with proxy_intercept_errors (see ticket #2033), moving to the next upstream server via proxy_next_upstream implies that nginx cannot maintain keepalive connection if the response body is not empty, since it does not try to read the body.

Closing this as a duplicate of #2033.

comment:2 by Maxim Dounin, 3 years ago

Resolution: duplicate
Status: newclosed
Note: See TracTickets for help on using tickets.