nginx serves stale 200 page which was deleted (became 404)

[devpets@…]

Hello I've read same bugs here but they were closed so I open a new one since I believe its not logic condition.

nginx relevant config:

proxy_cache_background_update on;
proxy_cache_key                "$MOBILE$scheme$host$request_uri";
proxy_cache_lock               on;
proxy_cache_methods            GET HEAD;
proxy_cache_use_stale          timeout invalid_header updating http_429 http_500 http_502 http_503 http_504 http_403; # 

proxy_cache_valid              200 15s;
proxy_cache_valid              404 15s;

---
I create simple file 404.txt and doing curl two times:

curl https://x.x.x/404.txt -k -I
HTTP/2 200 
server: nginx
date: Sun, 26 Nov 2023 07:44:29 GMT
content-type: text/plain; charset=UTF-8
content-length: 6
x-powered-by: WP Rocket/3.10.3
cache-control: public
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-nginx-upstream-cache-status: MISS
accept-ranges: bytes

again:

HTTP/2 200 
server: nginx
date: Sun, 26 Nov 2023 07:44:31 GMT
content-type: text/plain; charset=UTF-8
content-length: 6
x-powered-by: WP Rocket/3.10.3
cache-control: public
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-nginx-upstream-cache-status: HIT
accept-ranges: bytes

As you see now the file is cached in nginx.
Now I go and remove this file from server and doing curl again 4 times:

curl https://x.x.x/404.txt -k -I
HTTP/2 200 
server: nginx
date: Sun, 26 Nov 2023 07:45:27 GMT
content-type: text/plain; charset=UTF-8
content-length: 6
x-powered-by: WP Rocket/3.10.3
cache-control: public
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-nginx-upstream-cache-status: STALE
accept-ranges: bytes

HTTP/2 200 
server: nginx
date: Sun, 26 Nov 2023 07:45:29 GMT
content-type: text/plain; charset=UTF-8
content-length: 6
x-powered-by: WP Rocket/3.10.3
cache-control: public
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-nginx-upstream-cache-status: STALE
accept-ranges: bytes

HTTP/2 200 
server: nginx
date: Sun, 26 Nov 2023 07:46:34 GMT
content-type: text/plain; charset=UTF-8
content-length: 6
x-powered-by: WP Rocket/3.10.3
cache-control: public
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-nginx-upstream-cache-status: UPDATING
accept-ranges: bytes

HTTP/2 200 
server: nginx
date: Sun, 26 Nov 2023 07:46:48 GMT
content-type: text/plain; charset=UTF-8
content-length: 6
x-powered-by: WP Rocket/3.10.3
cache-control: public
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-nginx-upstream-cache-status: STALE
accept-ranges: bytes

nginx log:

10.15.1.15 - STALE [26/Nov/2023:10:25:24 +0200]  "HEAD /404.txt HTTP/2.0" 200 0 "-" "curl/7.74.0"

Now it stales forever, it never refresh.
I tried to add/remove this config:

proxy_cache_valid              404 15s;

but it doesn't change the behavior.
When I manually remove all cache files from nginx or if I add cache buster i.e:

curl https://x.x.x/404.txt?1=1

it begins to return 404.

10.15.1.15 - MISS [26/Nov/2023:10:27:05 +0200]  "HEAD /404.txt?1=1 HTTP/2.0" 404 0 "-" "curl/7.74.0"

Please advise how to proceed.
Dmitry Sherman

[Maxim Dounin]

As long as the response of the upstream server is not cacheable, nginx will not update the response stored in cache, and will continue to return the stale response from cache as per proxy_cache_use_stale updating; and proxy_cache_background_update in your configuration (see #853 for a feature request about somehow improving this behaviour).

With proxy_cache_valid 404 15s; 404 responses should be cacheable, though there might be other factors which prevents caching - such as Cache-Control: no-cache, Expires, Set-Cookie, or Vary: *, see ​proxy_cache_valid for details. The only 404 response shown is MISS, which suggests it is indeed non-cacheable regardless of the proxy_cache_valid 404. Check the 404 response headers to find out what prevents caching.

[devpets@…]

Hello thanks, this is the header of 404:
curl ​https://x.x.x/404.txt -k -I
HTTP/2 404
server: nginx
date: Mon, 27 Nov 2023 09:19:10 GMT
content-type: text/html; charset=UTF-8
vary: Accept-Encoding
expires: Wed, 11 Jan 1984 05:00:00 GMT
cache-control: no-cache, must-revalidate, max-age=0
x-xss-protection: 1; mode=block
x-content-type-options: nosniff

[Maxim Dounin]

Thanks for the confirmation. Exactly as expected, in your 404 responses caching is explicitly disabled with the Expires and Cache-Control headers:

expires: Wed, 11 Jan 1984 05:00:00 GMT
cache-control: no-cache, must-revalidate, max-age=0

If you want 404 to be cached, consider either reconfiguring your backend to avoid sending Expires and Cache-Control to disable caching, or configure nginx to ignore these headers (see ​proxy_ignore_headers).

Closing this as a duplicate of #853.

[devpets@…]

Hello I removed the cache-control header from backend, but still the page is served when it's removed.
I get this:
HTTP/2 200
server: nginx
date: Sun, 07 Jan 2024 14:04:23 GMT
content-type: text/html; charset=UTF-8
content-length: 0
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-nginx-upstream-cache-status: HIT
HTTP/2 200

next is:
x-nginx-upstream-cache-status: STALE
next is:
x-nginx-upstream-cache-status: UPDATING

and back to HIT.

This is 404 response:

curl ​https://x.x.x.x/405.php -k -I

HTTP/2 404
server: nginx
date: Sun, 07 Jan 2024 14:05:04 GMT
content-type: text/html; charset=UTF-8
vary: Accept-Encoding
expires: Wed, 11 Jan 1984 05:00:00 GMT
x-xss-protection: 1; mode=block
x-content-type-options: nosniff

As you see no cache-control headers are set.

For 404.php which was removed from server it keep serving from cache.

This is my nginx conf:
proxy_cache engintron_dynamic;
proxy_cache_background_update on;
proxy_cache_key "$MOBILE$scheme$host$request_uri";
proxy_cache_lock on;
proxy_cache_methods GET HEAD;
proxy_cache_use_stale timeout invalid_header updating http_429 http_500 http_502 http_503 http_504 http_403 http_404;

proxy_cache_valid 200 15s;
proxy_cache_valid 404 15s;

[devpets@…]

removed proxy_cache_use_stale http_404;
and problem solved

[Maxim Dounin]

Using proxy_cache_use_stale http_404; effectively disables caching of 404 responses, as it basically says "hey, nginx, if the backend returns 404 and there is something in the cache, drop the backend response and return what you have in the cache, even if it's expired". That is, similarly to your previous configuration with caching explicitly disabled by backend responses, the behaviour is expected.

As you've already correctly found out, fix would be to remove proxy_cache_use_stale http_404;.