Opened 2 years ago

Last modified 2 years ago

#1408 new defect

Big file download via php-fpm on fast networks eats up memory

Reported by: mmattel@… Owned by:
Priority: minor Milestone:
Component: nginx-core Version: 1.13.x
Keywords: Cc:
uname -a: Linux xxxx 4.4.0-51-generic #72-Ubuntu SMP Thu Nov 24 18:29:54 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.13.6 built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.5) built with OpenSSL 1.0.2g 1 Mar 2016 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie' --add-module=/opt/nginx-1.13.6/debian/modules/ngx_cache_purge

Description

The issue and the (temporary) solution is described at:
https://github.com/owncloud/core/issues/29328
https://github.com/owncloud/core/pull/29400

Ubuntu 16.04 with PHP7.0

Change History (3)

comment:1 Changed 2 years ago by mdounin

Are you using fastcgi_buffers 64 4K; as in the configuration provided in the issue? If yes, please test with a lower number of buffers (at most 63, default 8 is fine).

Using 64 fastcgi_buffers as in the configuration provided will end up in total 65 buffers being used to read the response, and this makes it possible that ngx_readv_file() will have to allocate memory which will be only freed when closing the connection, see d1bde5c3c5d2.

comment:2 follow-up: Changed 2 years ago by mmattel@…

Yes buffers were set to fastcgi_buffers 64 4K;
Testing with fastcgi_buffers 63 4K; as you said 63 should be max, downloading went fine and no memory was eaten up.

Two points:
a.) The check in the changeset referenced is >64 and not >63 ...
b.) Add a documentation note about this limit in http://nginx.org/en/docs/http/ngx_http_fastcgi_module.html#fastcgi_buffers because many people do not know about this "silent" issue ...

comment:3 in reply to: ↑ 2 Changed 2 years ago by mdounin

Replying to mmattel@…:

a.) The check in the changeset referenced is >64 and not >63 ...

As explained above, this is because 64 fastcgi_buffers ends up in 65 buffers in total (the fastcgi_buffer is also used to read the body).

b.) Add a documentation note about this limit in http://nginx.org/en/docs/http/ngx_http_fastcgi_module.html#fastcgi_buffers because many people do not know about this "silent" issue ...

Adding a documentation note may make sense, yes. On the other hand, it is an implementation detail which vary between versions (and hopefully will be fully resolved in future versions).

Note: See TracTickets for help on using tickets.