Opened 18 months ago

Last modified 18 months ago

#969 new enhancement

proxy module does not honour proxy_max_temp_file_size on cacheable responses

Reported by: Stealthii@… Owned by:
Priority: minor Milestone:
Component: nginx-module Version: 1.9.x
Keywords: slice proxy cache temp Cc:
Sensitive: no
uname -a: Linux 1dcda8eab5c1 4.4.8-boot2docker #1 SMP Mon Apr 25 21:57:27 UTC 2016 x86_64 Linux
nginx -V: nginx version: nginx/1.9.15 built by gcc 5.3.0 (Alpine 5.3.0) built with OpenSSL 1.0.2g 1 Mar 2016 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_perl_module=dynamic --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-mail --with-mail_ssl_module --with-file-aio --with-http_v2_module --with-ipv6

Description

In the ngx_http_proxy module, there is a directive: proxy_max_temp_file_size - it is intended to limit the size of buffered files on disk. There is a caveat:

"This restriction does not apply to responses that will be cached or stored on disk."

http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_max_temp_file_size

Why is this the case? I have an issue when using the slice module - I'm proxying a large request, and Nginx sends smaller range requests to the proxied server, for ~16MB hunks of the resource. Sometimes the proxied server misbehaves, and responds with a ranged response from the offset, till the end of the very large file.

Nginx will continue to 'temporarily' buffer this file to disk, resulting in a blocked downstream request, and an ever increasing use of the temporary proxy storage. The upstream file is multiple terabytes - Nginx should honor this directive and terminate the upstream range response when it receives over the max size for buffered files.

Change History (1)

comment:1 Changed 18 months ago by mdounin

  • Priority changed from major to minor

The proxy_max_temp_file_size is intended to limit maximum size of disk-based buffer. But when nginx is going to cache the response, it needs full response to be written to disk - not to buffer something, but to store the response. And hence this restriction is ignored.

There are no plans to change the current behaviour, though we can consider adding an option to limit maximum size of responses to be cached. Right now you can use proxy_no_cache with appropriate checks of $upstream_http_content_length to avoid caching large responses (assuming Content-Length of a response is know in advance).

Note: See TracTickets for help on using tickets.