Nginx doesn't delete temp cache files after a crash

[ifel@…]

When Nginx cache module is downloading an object from an upstream, it saves it into a temp file, and once the download is finished, Nginx renames the temp file to a cache one. Nginx does not track these temp files in any persistent storage, nor Cache manager detects them. Thus, if Nginx crashes or gets killed when it's downloading objects, these temp files stays on disk and nothing cleans them up. Since most of modern linux uses systemd, it's a default behavior, if after sending TERM to a process, it does not finish within 60 sec, systemd sends KILL.

There should be some housekeeping mechanism cleaning these stale files up.

Example of these files, it was captured on May 12:
-rw------- 1 nginx nginx 52035584 May 10 12:06 d43aaa174b966c213a8b65a8d53e8c01.0000935649
-rw------- 1 nginx nginx 52117504 May 10 12:06 d43aaa174b966c213a8b65a8d53e8c01.0000935663
-rw------- 1 nginx nginx 52592640 May 10 12:06 d43aaa174b966c213a8b65a8d53e8c01.0000935679
-rw------- 1 nginx nginx 52248576 May 10 12:06 d43aaa174b966c213a8b65a8d53e8c01.0000935687
-rw------- 1 nginx nginx 41664512 May 10 12:06 d43aaa174b966c213a8b65a8d53e8c01.0000935688
-rw------- 1 nginx nginx 45858816 May 10 12:06 d43aaa174b966c213a8b65a8d53e8c01.0000935701
-rw------- 1 nginx nginx 41746432 May 10 12:06 d43aaa174b966c213a8b65a8d53e8c01.0000935705
-rw------- 1 nginx nginx 42139648 May 10 12:06 d43aaa174b966c213a8b65a8d53e8c01.0000935708
-rw------- 1 nginx nginx 37421056 May 10 12:06 d43aaa174b966c213a8b65a8d53e8c01.0000935714
-rw------- 1 nginx nginx 36159488 May 10 12:06 d43aaa174b966c213a8b65a8d53e8c01.0000935720
-rw------- 1 nginx nginx 36683776 May 10 12:06 d43aaa174b966c213a8b65a8d53e8c01.0000935730
-rw------- 1 nginx nginx 36847616 May 10 12:06 d43aaa174b966c213a8b65a8d53e8c01.0000935734
-rw------- 1 nginx nginx 40304640 May 10 12:06 d43aaa174b966c213a8b65a8d53e8c01.0000935739
-rw------- 1 nginx nginx 24592384 May 10 12:06 d43aaa174b966c213a8b65a8d53e8c01.0000935743
-rw------- 1 nginx nginx 27017216 May 10 12:06 d43aaa174b966c213a8b65a8d53e8c01.0000935746
-rw------- 1 nginx nginx 30097408 May 10 12:06 d43aaa174b966c213a8b65a8d53e8c01.0000935754

[Maxim Dounin]

While it would be nice to have something to remove stale temporary cache files, it is generally not possible to do this automatically, since these files might be actually used by running nginx processes. Current recommendation is to remove such files manually if needed.

Note that it is generally a bad idea to abruptly stop nginx. Instead, consider using ​configuration reload and ​binary upgrade. And, of course, if nginx crashes, you should investigate why it does so, and resolve the underlying issue.

[ifel@…]

I understand that it's better to use reload. But there are some cases when it needs to be stopped. Normally, this is done by a supervisor, on modern OS it's systemd, and it's default behavior of systemd - run a command specified in ExecStop=, otherwise send SIGTERM, wait for TimeoutStopSec= seconds, default 90s, and if the processes are still running send SIGKILL.

So, if the process does not stop within this timeout it will be killed. Of course it's possible to increase the timeout, but it does not guarantee it will work in all the cases, and it will not prevent from appearing stale files in other situations like bad/buggy supervisor, or OOM killer.

I don't know if the cache manager has access to processes, but in theory it can be possible to have the threads store file names they are working with in a shared memory and have the cache manager reading ctime/mtime of the files to avoid race condition, something like this:

cur_time = now()
active_temp_fns = read_fns_from_memory()
fns_on_disk = read_fn_from_disk()
for f in fns_on_disk:

if f in active_temp_fns:

continue

if stat(f).mtime < cur_time:

rm(f)

This would've made it cleaning after itself despite any external processes/users killing the process.

[Maxim Dounin]

I don't know if the cache manager has access to processes, but in theory it can be possible to have the threads store file names they are working with in a shared memory

This is not going to work, as during binary upgrade there is no shared memory between old and new processes. Not to mention the overhead this approach implies.