Opened 4 weeks ago

Closed 3 weeks ago

#1888 closed defect (invalid)

Nginx cache has multiple formats, and many of them are invalid during cleaning, so it is impossible to clean all of them

Reported by: yunoasgit@… Owned by:
Priority: minor Milestone:
Component: other Version: 1.17.x
Keywords: Cc:
uname -a:
nginx -V: nginx/1.17.3

Description

Nginx cache has multiple formats, and many of them are invalid during cleaning, so it is impossible to clean all of them
Nginx / 1.17.3 cleaning cache is invalid,
The same purge method, no problem compared with nginx / 1.16.0, normal cleaning.

proxy_ignore_headers Accept-Encoding ;

nginx version: nginx/1.17.3
conf:


proxy_cache_path /cache/www levels=1:2 keys_zone=wwwcache:500m inactive=7d max_size=1000m;

location ~ .*\.(xml|htm|html|css|js)(.*) {

proxy_pass http://www.xxx.com;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header Connection "";
proxy_set_header Accept-Encoding ;
proxy_http_version 1.1;
client_body_buffer_size 128k;
proxy_connect_timeout 60;
proxy_send_timeout 60;
proxy_read_timeout 60;
proxy_buffer_size 32k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
proxy_cache wwwcache;
proxy_cache_valid 200 302 30m;
proxy_cache_valid 301 30m;
proxy_cache_valid any 1m;
proxy_cache_key $host$uri$is_args$args;

}


test run
#1
curl -o /dev/null "http://www.xxx.com/common/js/jquery.min.js" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit?/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36" -x 10.1.2.3:80

curl -o /dev/null 'http://www.xxx.com/common/js/jquery.min.js' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit?/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36' -H 'accept-encoding: gzip, deflate, br' -x 10.1.2.3:80

curl -o /dev/null 'http://www.xxx.com/common/js/jquery.min.js' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit?/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36' -H 'accept-encoding: gzip, deflate' -x 10.1.2.3:80

curl -o /dev/null 'http://www.xxx.com/common/js/jquery.min.js' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit?/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36' -H 'accept-encoding: gzip' -x 10.1.2.3:80

curl -o /dev/null 'http://www.xxx.com/common/js/jquery.min.js' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit?/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36' -H 'accept-encoding: deflate, gzip' -x 10.1.2.3:80

curl -o /dev/null 'http://www.xxx.com/common/js/jquery.min.js' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit?/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36' -H 'accept-encoding: deflate, gzip, br ' -x 10.1.2.3:80


After six accesses, six cache files are generated


./6/06/6ba389ade4e2d55d4e66fbb8d99e5066
./0/70/8df2f99ae842a8d7f5d5192da6e39700
./e/dd/2db0e8eef68e05aaa0b9fbb01a7dbdde
./d/09/b829dd1da559e3c80aeaabf58374c09d
./a/c3/547655366b88299921b477dbeeffdc3a
./a/72/61a3eac721aa5d15a14b64630d89172a


Clean cache in turn
#2
curl -o /dev/null "http://www.xxx.com/purge/common/js/jquery.min.js" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit?/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36" -x 10.1.2.3:80

curl -o /dev/null 'http://www.xxx.com/purge/common/js/jquery.min.js' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit?/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36' -H 'accept-encoding: gzip, deflate, br' -x 10.1.2.3:80

curl -o /dev/null 'http://www.xxx.com/purge/common/js/jquery.min.js' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit?/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36' -H 'accept-encoding: gzip, deflate' -x 10.1.2.3:80

curl -o /dev/null 'http://www.xxx.com/purge/common/js/jquery.min.js' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit?/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36' -H 'accept-encoding: gzip' -x 10.1.2.3:80

curl -o /dev/null 'http://www.xxx.com/purge/common/js/jquery.min.js' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit?/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36' -H 'accept-encoding: deflate, gzip' -x 10.1.2.3:80

curl -o /dev/null 'http://www.xxx.com/purge/common/js/jquery.min.js' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit?/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36' -H 'accept-encoding: deflate, gzip, br ' -x 10.1.2.3:80


Only one cache was successfully cleaned up, and the other five failed to clean up the cache properly.

ls: cannot access ./a/72/61a3eac721aa5d15a14b64630d89172a: No such file or directory


The other five are still on disk

./6/06/6ba389ade4e2d55d4e66fbb8d99e5066
./0/70/8df2f99ae842a8d7f5d5192da6e39700
./e/dd/2db0e8eef68e05aaa0b9fbb01a7dbdde
./d/09/b829dd1da559e3c80aeaabf58374c09d
./a/c3/547655366b88299921b477dbeeffdc3a


Nginx cleans up cache bugs

Thank you very much.

Change History (2)

comment:1 Changed 4 weeks ago by yunoasgit@…

After many attempts, the temporary solution to the bug is,

  1. Request no "accept encoding: gzip" again to hit it to generate the cache file
  2. Then clean up 'accept encoding: deflate' and 5 hit files first
  3. Last clean up "no gzip"

In this way, all 6 cache files can be cleaned up

comment:2 Changed 3 weeks ago by mdounin

  • Resolution set to invalid
  • Status changed from new to closed

As long as upstream responses use Vary response header, multiple cache files can be created. Quoting docs:

If the header includes the “Vary” field with another value, such a response will be cached taking into account the corresponding request header fields (1.7.7).

Manually removing all relevant files from the filesystem might be tricky if there are multiple variants. This is not something you are expected to do though. Instead, consider using existing cache validity and cleanup mechanisms, such as Cache-Control, proxy_cache_valid, and proxy_cache_path parameters inactive and max_size.

If you want to disable Vary support, you can do so using the proxy_ignore_headers directive. Alternatively, you can change your backend to avoid generating Vary.

Note: See TracTickets for help on using tickets.