Opened 6 years ago

Last modified 5 years ago

#1579 new defect

Mirror subrequests ignore the keepalive flag

Reported by: predatory.kangaroo@… Owned by:
Priority: minor Milestone:
Component: other Version: 1.14.x
Keywords: Cc:
uname -a: Linux z-space 4.13.0-43-generic #48~16.04.1-Ubuntu SMP Thu May 17 12:56:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.14.0
built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.4)
built with OpenSSL 1.0.1f 6 Jan 2014 (running with OpenSSL 1.0.2g 1 Mar 2016)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

Description

I'm attempting to use the mirror directive to soak test a new backend service, and am experiencing issues due to the upstream connections being closed after each mirror subrequest. This makes the test impossible, as we hit port exhaustion in < 60s.

I've built a test case using a simple Go server and ab for load testing, attached as test-site.conf and run-test.sh. This test case demonstrates that keepalive functionality works fine for the upstream, except in the case of mirror (and auth_request) subrequests.

I've also done some source diving, and discovered that this can be worked around by changing ngx_http_mirror_module.c:171 to set sr->discard_body = 1 instead of sr->header_only = 1, though I have no idea what side effects this might cause.

Attachments (2)

test-site.conf (1.4 KB ) - added by predatory.kangaroo@… 6 years ago.
Test site config
run-tests.sh (420 bytes ) - added by predatory.kangaroo@… 6 years ago.
Test runner

Download all attachments as: .zip

Change History (4)

by predatory.kangaroo@…, 6 years ago

Attachment: test-site.conf added

Test site config

by predatory.kangaroo@…, 6 years ago

Attachment: run-tests.sh added

Test runner

comment:1 by Roman Arutyunyan, 6 years ago

A mirror subrequest response body is useless both for client output and for saving in memory. That's why the header_only flag is set for a mirror subrequest. The flag triggers some optimizations which may disable upstream keepalive as a side-effect.

This is not the only case when upstream keepalive gets disabled as a result of optimization. Another case is when upstream response comes before request body is fully sent. In most cases this works fine since establishing a new connection is usually faster than trying to send or read useless data. But probably we need a directive to force upstream connection consistency (send all and read all), which will enable keepalive for special cases. Currently we have no specific plans about this issue.

A simple and dirty workaround for your case is enabling proxy_cache or proxy_store.

comment:2 by alonana@…, 5 years ago

Actually, I've encountered this issue, and was able to bypass the auth keep alive disable issue, by using the NGINX JavaScript. See this post https://runkiss.blogspot.com/2020/03/nginx-performance-tuning.html for full details

Note: See TracTickets for help on using tickets.