id summary reporter owner description type status priority milestone component version resolution keywords cc uname nginx_version 1163 cache size grows over max_size turchanov@… "Periodically we have proxy_cache size growing over the configured limit (see attached graph, mind Zabbix 150G=140GiB) which usually after several hours gets back to normal. I did some gdb poking in the 'cache manager' process and it shows that '''ngx_http_file_cache_manager''' is called and the current cache '''size''' is indeed larger than '''max_size''' {{{ (gdb) 1940 size = cache->sh->size; (gdb) print size $2 = 42999201 (gdb) print cache->max_size $3 = 36700160 }}} but '''ngx_http_file_cache_forced_expire''' deletes nothing due to locked cache entries (fcn->count == 1) and after hard-coded limit of 20 the search is aborted. #749 suggests nginx worker crashes which we had none. But I did kill with TERM signal several times after reconfigure workers in '''worker process is shutting down''' state as I needed to suppress activity of workers with previous configuration as soon as possible. So the question is could termination of workers with TERM signal cause the issue and if so why didn't workers perform cleanup of holding locks in caches? Or could it be '''proxy_cache_lock on''' which causes trouble? And there is one thing that bothers we: why the cache manager eventually succeeds in cleaning up the cache (as you can see on the attached graph)? If it were locked entries that left hanging after killed worker process with TERM signal then the cache manager wouldn't be able to clean up. My config (I hope that I didn't cut too much): {{{ http { ... proxy_cache_path /opt/cache/static levels=1:2 use_temp_path=off keys_zone=static:2048m inactive=1y max_size=140g manager_files=50000 manager_sleep=1 manager_threshold=1000 loader_files=50000 loader_threshold=1000 loader_sleep=1; ... server { location / { proxy_cache static; proxy_no_cache $no_cache_static; proxy_cache_bypass $no_cache_static; proxy_cache_lock on; proxy_cache_lock_timeout 1s; proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504; proxy_cache_valid 200 1y; proxy_pass http://static_nodes; } } } }}}" defect closed critical nginx-core 1.11.x worksforme Linux 4.1.12-61.1.19.el7uek.x86_64 #2 SMP Fri Nov 18 17:00:09 PST 2016 x86_64 x86_64 x86_64 GNU/Linux "nginx version: nginx/1.11.7 built by gcc 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) built with OpenSSL 1.0.2j-fips 26 Sep 2016 TLS SNI support enabled configure arguments: --prefix=/opt/nginx-1.11.7 --pid-path=/var/run/nginx.pid --http-client-body-temp-path=/tmp/nginx-client_body_temp --http-proxy-temp-path=/tmp/nginx-proxy_temp --http-fastcgi-temp-path=/tmp/nginx-fastcgi_temp --http-uwsgi-temp-path=/tmp/nginx-uwsgi_temp --http-scgi-temp-path=/tmp/nginx-scgi_temp --with-pcre --with-http_ssl_module --with-openssl=/opt/orabuild/rpmbuild/BUILD/nginx-1.11.7-addons/openssl-1.0.2j --with-openssl-opt='no-krb5 enable-camellia enable-seed enable-tlsext enable-rfc3779 no-mdc2 no-ec2m no-gost no-srp -fno-strict-aliasing fips' --with-http_v2_module --with-http_stub_status_module --with-http_gzip_static_module --with-http_gunzip_module --with-http_realip_module --with-stream --with-stream_ssl_module --with-http_geoip_module=dynamic --add-module=/opt/orabuild/rpmbuild/BUILD/nginx-1.11.7-addons/ngx_devel_kit-master --add-dynamic-module=/opt/orabuild/rpmbuild/BUILD/nginx-1.11.7-addons/lua-nginx-module-master --add-dynamic-module=/opt/orabuild/rpmbuild/BUILD/nginx-1.11.7-addons/lua-upstream-nginx-module-master --add-dynamic-module=/opt/orabuild/rpmbuild/BUILD/nginx-1.11.7-addons/set-misc-nginx-module-master --add-dynamic-module=/opt/orabuild/rpmbuild/BUILD/nginx-1.11.7-addons/array-var-nginx-module-master --add-module=/opt/orabuild/rpmbuild/BUILD/nginx-1.11.7-addons/echo-nginx-module-0.60 --add-dynamic-module=/opt/orabuild/rpmbuild/BUILD/nginx-1.11.7-addons/nginx-module-vts-0.1.11 --add-module=/opt/orabuild/rpmbuild/BUILD/nginx-1.11.7-addons/nginx_upstream_check_module-master --add-dynamic-module=/opt/orabuild/rpmbuild/BUILD/nginx-1.11.7-addons/nchan-1.0.8 "