open_file_cache loses trailing newline on changed files

[Sam Bull]

When using open_file_cache, if I modify a file, the server starts serving the new file without a trailing newline until the cache expires.

This causes issues with SRI, where the hash of the file (which includes a trailing newline) is created by the application and included in the integrity attribute. Because Nginx is serving a version of the file to the browser without the newline, this causes the SRI check to fail and the browser refuses to run the resource.

[Maxim Dounin]

If you non-atomically modify a file being served by a web server, you create a race condition: if nginx opens a file and starts sending it before the modification, and continues sending after the modification, the client will receive a mix of the old and modified files. Using open_files_cache ensures that this race condition will happen in practice even under low load, as files are opened and stat()'ed long before they are actually sent to the client.

When the file is make smaller during a in-place modification, nginx is likely to notice this, and will log an pread() read only ... of ... alert as in #139, or sendfile() reported that ... was truncated at ... alert when using sendfile(). However, if the file is made bigger or the size is preserved, it is not possible to automatically detect this without additional checks (or at all), so you are likely to only observe the corruption when looking into the response contents.

The only proper solution is to avoid non-atomic file modifications. That is, you have to update files atomically instead: write a new file with a temporary name, and then use mv to atomically rename it to the correct name. With this approach, nginx will be able to continue serving the old file till it is no longer needed, eliminating the race condition.

If you are not able to use atomic file updates for some reason, consider switching open_file_cache off - this will make the race condition created by the non-atomic modifications less likely to manifest itself in practice. Note though that this won't eliminate the race condition, and it is still possible that a corrupted response will be returned if a modification happens in the wrong time. As outlined above, the only proper solution is to use atomic file updates instead.

[Sam Bull]

OK, but would it also be possible to use something like inotify to get alerted when a file has changed and clear the modified file from the cache?

Expecting users over SFTP to create temporary files and then rename them seems a little too much, plus getting applications (like a CMS) to do the same when updating files seems extremely difficult.

I understand there is still a chance of the race condition occurring, but atleast with the inotify solution it would work as expected the vast majority of the time, rather than often having unexpected results as the current solution provides.

[Maxim Dounin]

OK, but would it also be possible to use something like inotify to get alerted when a file has changed and clear the modified file from the cache?

There were some attempts in the past to introduce event-based invalidation in open_file_cache (activated via the open_file_cache_events directive, uses kqueue's EVFILT_VNODE). This work was never complete though, mostly because the open_file_cache itself provides little-to-no benefits except may be in some very specific workloads.

If you cannot enforce proper atomic file updates in your environment, just switch open_file_cache off, this will minimize possible damage from non-atomic file modifications.