Stale 200 Served even when backend sets 404 with new proxy_cache_background_update directive

[stackoverflow.com/users/2528298/ted-wilmont]

Hi there,

We have recently enabled proxy_cache_background_update and what a great feature it is.

However, we have noticed a bug. Hopefully it's just a mis-config on our part but I thought I'd post it as I can't seem to find out why this is happening.

We run a backend server that sets X-Accel-Expires headers at 3600 seconds.

The problem we had was when we enabled proxy_cache_background_update, if we deleted a page on the origin server that was previously cached it would continuously serve the old stale 200 page even though the backend was returning a 404 - even after 3600 seconds.

The 200 response was always "STALE" and never updated as a 404. To fix this, we had to disable proxy_cache_background_update. Once that was disabled, a correct 404 page was sent to the client.

Our config (in http context):

proxy_cache_use_stale  updating;
proxy_cache_background_update on;

As you can see, we are not using "proxy_cache_use_stale error;" or anything like that; just "updating".

Our proxy cache config:

proxy_cache_path /cache levels=1:2 keys_zone=one:256m max_size=128g inactive=14d;
proxy_cache_key $scheme$host$request_uri;

Our config for the site in question:

server {

        proxy_cache one;
    listen 1.1.1.1:443 ssl http2;
    server_name www.oursite.co.uk oursite.co.uk;

    ssl_certificate           /certificates/www.oursite.co.uk.cer;
    ssl_certificate_key       /certificates/www.oursite.co.uk.key;

    ssl on;
    ssl_session_cache  builtin:1000  shared:SSL:10m;
    ssl_dhparam /certificates/dhparams.pem;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
    ssl_prefer_server_ciphers on;



#ssl_stapling on;
#ssl_stapling_verify on;
#ssl_trusted_certificate /certificates/thawte.cer;
    resolver 8.8.4.4 8.8.8.8 valid=300s;
    resolver_timeout 10s;


location / {


set $wordpress_auth "";
    if ($http_cookie ~* "wordpress_logged_in_[^=]*=([^%]+)%7C") {
        set $wordpress_auth wordpress_logged_in_$1;
    }
        proxy_cache_bypass $wordpress_auth;
        proxy_no_cache $wordpress_auth;
        #add_header Strict-Transport-Security "max-age=31536000";
        add_header X-Cache $upstream_cache_status;
        proxy_set_header        Host $host;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $remote_addr;
        proxy_set_header        X-Forwarded-Proto $scheme;
        proxy_pass          http://1.1.1.1;
        proxy_read_timeout  180s;
}
  }

Hopefully somebody can shed some light?

[stackoverflow.com/users/2528298/ted-wilmont]

Update:

The original page had the X-Accel-Expires header set at 3600 seconds but the new 404 on the same URL had no X-Accel-Expires headers set at all with an expires header set at: Wed, 11 Jan 1984 05:00:00 GMT.

[Roman Arutyunyan]

The background cache updater receives an uncacheable response (expires in the past) and ignores it. What behaviour did you expect?

[stackoverflow.com/users/2528298/ted-wilmont]

But that's not the case when proxy_cache_background_update is disabled. When proxy_cache_background_update is disabled, the correct 404 page is sent to the client and the cache entry is removed - as expected when 3600 seconds have expired.

I would expect the STALE page to be removed and to forward on the correct 404.

Once the 3600 seconds have expired, the background cache updater will fetch the URL once again from the origin server and if it 404s it should then forward that 404 on to future clients. That's how I would anticipate it should work. It works this way with proxy_cache_background_update disabled.

Maybe I'm not configuring correctly? How can changes to the config perform this behaviour?

[Roman Arutyunyan]

When proxy_cache_background_update is disabled, there's a client, which receives the 404 response (which is not cached anyway). In the background subrequest there's no client to send 404 to. Why would nginx cache an already expired response? In a way, it's even "more stale" than the old one.

[stackoverflow.com/users/2528298/ted-wilmont]

Thank you for your reply.

In that case, how do we configure nginx to drop the cache entry when the background updater receives a 404? Otherwise, nginx would never forward the 404 for URLs that have been removed (but were once previously cached)?

[Roman Arutyunyan]

What you can do is disable getting cache time from the response using the "proxy_ignore_headers" directive. And instead set it directly with "proxy_cache_valid".

proxy_ignore_headers Expires;
proxy_cache_valid 404 1s;

[stackoverflow.com/users/2528298/ted-wilmont]

That's interesting, and thank you for suggesting that.

However, we also cache image/video files from the origin server and deliver them using nginx direct to the client. These have expires headers of 3 months set on the origin server. We don't want to set 3 months on all "200" codes so how could we work around that?

[Maxim Dounin]

Closing this, observed behaviour is as intended and not a bug. It is also not really related to proxy_cache_background_update, but rather a result of proxy_cache_use_stale used. Background updates just make the behaviour more obvious and easier to observe.

There are two basic options to resolve this:

• Make sure that resources cached do not return uncacheable results to revalidation requests, as suggested by Roman.
• Avoid using proxy_cache_use_stale updating for infinite time, but use Cache-Control: stale-while-revalidate=<seconds> header instead. This allows to specify maximum time after which a stale response will be removed.