Context Navigation

← Previous Ticket
Next Ticket →

#1888 closed defect (invalid)

Nginx cache has multiple formats, and many of them are invalid during cleaning, so it is impossible to clean all of them

Reported by:	yunoasgit@…	Owned by:
Priority:	minor	Milestone:
Component:	other	Version:	1.17.x
Keywords:		Cc:
uname -a:
nginx -V:	nginx/1.17.3

Description

Nginx cache has multiple formats, and many of them are invalid during cleaning, so it is impossible to clean all of them
Nginx / 1.17.3 cleaning cache is invalid,
The same purge method, no problem compared with nginx / 1.16.0, normal cleaning.

proxy_ignore_headers Accept-Encoding ;

nginx version: nginx/1.17.3
conf:

proxy_cache_path /cache/www levels=1:2 keys_zone=wwwcache:500m inactive=7d max_size=1000m;

location ~ .*\.(xml|htm|html|css|js)(.*) {

proxy_pass http://www.xxx.com;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header Connection "";
proxy_set_header Accept-Encoding ;
proxy_http_version 1.1;
client_body_buffer_size 128k;
proxy_connect_timeout 60;
proxy_send_timeout 60;
proxy_read_timeout 60;
proxy_buffer_size 32k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
proxy_cache wwwcache;
proxy_cache_valid 200 302 30m;
proxy_cache_valid 301 30m;
proxy_cache_valid any 1m;
proxy_cache_key $host$uri$is_args$args;

}

test run
#1
curl -o /dev/null "http://www.xxx.com/common/js/jquery.min.js" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36" -x 10.1.2.3:80

curl -o /dev/null 'http://www.xxx.com/common/js/jquery.min.js' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36' -H 'accept-encoding: gzip, deflate, br' -x 10.1.2.3:80

After six accesses, six cache files are generated

./6/06/6ba389ade4e2d55d4e66fbb8d99e5066
./0/70/8df2f99ae842a8d7f5d5192da6e39700
./e/dd/2db0e8eef68e05aaa0b9fbb01a7dbdde
./d/09/b829dd1da559e3c80aeaabf58374c09d
./a/c3/547655366b88299921b477dbeeffdc3a
./a/72/61a3eac721aa5d15a14b64630d89172a

Clean cache in turn
#2
curl -o /dev/null "http://www.xxx.com/purge/common/js/jquery.min.js" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36" -x 10.1.2.3:80

curl -o /dev/null 'http://www.xxx.com/purge/common/js/jquery.min.js' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36' -H 'accept-encoding: gzip, deflate, br' -x 10.1.2.3:80

Only one cache was successfully cleaned up, and the other five failed to clean up the cache properly.

ls: cannot access ./a/72/61a3eac721aa5d15a14b64630d89172a: No such file or directory

The other five are still on disk

./6/06/6ba389ade4e2d55d4e66fbb8d99e5066
./0/70/8df2f99ae842a8d7f5d5192da6e39700
./e/dd/2db0e8eef68e05aaa0b9fbb01a7dbdde
./d/09/b829dd1da559e3c80aeaabf58374c09d
./a/c3/547655366b88299921b477dbeeffdc3a

Nginx cleans up cache bugs

Thank you very much.

Change History (7)

comment:1 by yunoasgit@…, 5 years ago

After many attempts, the temporary solution to the bug is,

Request no "accept encoding: gzip" again to hit it to generate the cache file
Then clean up 'accept encoding: deflate' and 5 hit files first
Last clean up "no gzip"

In this way, all 6 cache files can be cleaned up

comment:2 by Maxim Dounin, 5 years ago

Resolution:	→ invalid
Status:	new → closed

As long as upstream responses use Vary response header, multiple cache files can be created. Quoting docs:

If the header includes the “Vary” field with another value, such a response will be cached taking into account the corresponding request header fields (1.7.7).

Manually removing all relevant files from the filesystem might be tricky if there are multiple variants. This is not something you are expected to do though. Instead, consider using existing cache validity and cleanup mechanisms, such as Cache-Control, proxy_cache_valid, and proxy_cache_path parameters inactive and max_size.

If you want to disable Vary support, you can do so using the proxy_ignore_headers directive. Alternatively, you can change your backend to avoid generating Vary.

comment:3 by yunoasgit@…, 5 years ago

Nginx / 1.16.0 does not have this bug,

Do you want to fix the bug in the later version?

follow-up: 5 comment:4 by Maxim Dounin, 5 years ago

Nginx / 1.16.0 does not have this bug

Caching behaviour in 1.16.0 is the same as in nginx 1.17.x. Either way, the bug is in your cleaning code, not in nginx, see comment:2 for details.

in reply to: 4 comment:5 by yunoasgit@…, 5 years ago

Replying to Maxim Dounin:

Nginx / 1.16.0 does not have this bug

Caching behaviour in 1.16.0 is the same as in nginx 1.17.x. Either way, the bug is in your cleaning code, not in nginx, see comment:2 for details.

In order to verify, this is a direct RM deletion

Installation sequence execution

#test1

curls -v "http://m.xxxxxx.com/yy/static/js/play/playbyAudio.js" -H "Host: m.xxxxxx.com" -H "Accept-Encoding: gzip, deflate" -x 10.10.10.72:80

find View changes, generate files ./e/47/2fbd0a5ddf085e5d4fc97f8a2131547e

#test2

curls -v "http://m.xxxxxx.com/yy/static/js/play/playbyAudio.js" -H "Host: m.xxxxxx.com" -x 10.10.10.72:80

find View changes, generate files ./7/99/b5d500b076b42ab1b8082fd570d13997
#test3

curls -v "http://m.xxxxxx.com/yy/static/js/play/playbyAudio.js" -H "Host: m.xxxxxx.com" -H "Accept-Encoding: gzip, deflate, sdch" -x 10.10.10.72:80

find View changes, generate files ./5/be/7033df66894e639097a71136d9dc7be5

#del
First request, file generated by test1 curl
rm -f ./e/47/2fbd0a5ddf085e5d4fc97f8a2131547e

#test2

curls -v "http://m.xxxxxx.com/yy/static/js/play/playbyAudio.js" -H "Host: m.xxxxxx.com" -x 10.10.10.72:80

find View changes, generate files ./e/47/2fbd0a5ddf085e5d4fc97f8a2131547e
But documents ./7/99/b5d500b076b42ab1b8082fd570d13997 It exists. It's not hit

#test1

curls -v "http://m.xxxxxx.com/yy/static/js/play/playbyAudio.js" -H "Host: m.xxxxxx.com" -H "Accept-Encoding: gzip, deflate" -x 10.10.10.72:80

find View changes, generate files ./2/e4/e531e80345b26049165ef398ef2d5e42 It's not hit

#del，Continue to delete it
rm -f ./e/47/2fbd0a5ddf085e5d4fc97f8a2131547e

#test1

curls -v "http://m.xxxxxx.com/yy/static/js/play/playbyAudio.js" -H "Host: m.xxxxxx.com" -H "Accept-Encoding: gzip, deflate" -x 10.10.10.72:80

find View changes, generate files ./e/47/2fbd0a5ddf085e5d4fc97f8a2131547e

#test3

curls -v "http://m.xxxxxx.com/yy/static/js/play/playbyAudio.js" -H "Host: m.xxxxxx.com" -H "Accept-Encoding: gzip, deflate, sdch" -x 10.10.10.72:80

HIT ./5/be/7033df66894e639097a71136d9dc7be5

find
./2/e4/e531e80345b26049165ef398ef2d5e42
./5/be/7033df66894e639097a71136d9dc7be5
./7/99/b5d500b076b42ab1b8082fd570d13997
./e/47/2fbd0a5ddf085e5d4fc97f8a2131547e
A total of 4 cache files have been generated (the cat file has confirmed that the URL is correct)
There are only three cases of curl test
Verify the same result multiple times

Report that it has such problems
Want to fix it

Last edited 5 years ago by yunoasgit@… (previous) (diff)

comment:6 by Maxim Dounin, 5 years ago

The observed behaviour is expected.

As long as there is no main cache file (the one generated based on the proxy_cache_key), nginx will use it to store the response. Secondary keys are used for additional variants as long as the main cache file already exists. Further, the main cache file is used to store Vary information, so it is not possible to calculate secondary keys without the main cache file in place.

Removing the main cache file as in your tests will result in the main cache file being recreated. The new main cache file will contain the variant requested, regardless of whether this variant was previously cached under a secondary key. If the variant requested was previously cached under a secondary key, the now unused secondary cache file will be eventually removed based on inactive= and/or max_size= parameters of the proxy_cache_path directive.

comment:7 by yunoasgit@…, 5 years ago

It is hoped that the cache file can be deleted through del, and key will not change and will not confuse key and cache.
Key and cache
One-to-one correspondence, no confusion

Thank you very much for your support.

Note: See TracTickets for help on using tickets.

Download in other formats: