proxy_cache bug when using custom error pages

[stackoverflow.com/users/220086/denis-pshenov]

Hi,

I have a very simple proxy config:

http {

proxy_cache_path /var/www/cache levels=1:2 keys_zone=s3-images-cache:50m inactive=1M max_size=1000m;
proxy_temp_path /var/www/cache/tmp;

server {

listen 80;
server_name images.example.com;

location / {

proxy_cache s3-images-cache;
proxy_cache_key $scheme$proxy_host$uri$is_args$args;
proxy_cache_bypass $http_purge_cache;
proxy_cache_valid any 1y;
proxy_pass ​http://images-example.s3.amazonaws.com;
add_header X-Cache $upstream_cache_status;

proxy_intercept_errors on;
error_page 404 = @no_image;

}

location @no_image {

return 403;

}

}

}

Now follow me here:

1. Let's request /image.jpg.
2. Request is sent to proxy for /image.jpg (does not exists yet).
3. Backend responds with 404.
4. "proxy_intercept_errors on" kicks in and "error_page 404 = @no_image" is called.
5. Nginx returns 403.
6. Do another request for same image and see that "X-Cache: HIT" is set. We are clearly hitting cache.

But, if we check /var/www/cache/ folder at this time we will see that there is no cache item created for this request. So does it mean Nginx keeps the cache for it in memory and forgot to write to file?

7. Let's upload /image.jpg to backend.
8. Now do "PURGE-CACHE: 1" request to that image. We see that now we get the image instead of 403. Good.

If we check /var/www/cache/ we will see that the cache file is now finally created for this request. Also good.

9. Now here is the problem: lets request /image.jpg again.
10. Nginx returns 403 with "X-Cache: HIT". Why? So it's hitting the cache but returning something else not what is in /var/www/cache folder?? How?

My only explanation to this is it seems that Nginx is caching the response in memory and doesn't write to file when we are hitting error with our custom error_page in the proxied responses. Furthermore when using proxy_cache_bypass it does not overwrite in-memory cache, so that subsequent requests to the same item will be using old cache which is stored in memory and not the new one created in the cache folder.

[Maxim Dounin]

When caching errors which are intercepted using proxy_intercept_errors, nginx only stores error code in memory (and nothing more). That's why you see cache used but no cache file at step 6.

As for the errors returned again at step 10, it looks like a bug related to proxy_cache_bypass. The following patch should fix it:

# HG changeset patch
# User Maxim Dounin <mdounin@mdounin.ru>
# Date 1463067394 -10800
#      Thu May 12 18:36:34 2016 +0300
# Node ID 26a9444d07f49cd8b3c0fbc4e167999b4ef45f8e
# Parent  3aed80d49dd8eb32482ce2d277ddd2b3d77b7363
Cache: fixed updating bypassed cached errors (ticket #827).

diff --git a/src/http/ngx_http_file_cache.c b/src/http/ngx_http_file_cache.c
--- a/src/http/ngx_http_file_cache.c
+++ b/src/http/ngx_http_file_cache.c
@@ -1403,6 +1403,7 @@ ngx_http_file_cache_update(ngx_http_requ
     ngx_shmtx_lock(&cache->shpool->mutex);
 
     c->node->count--;
+    c->node->error = 0;
     c->node->uniq = uniq;
     c->node->body_start = c->body_start;
 

[Maxim Dounin <mdounin@…>]

In 6539:d94f57990492/nginx:

Cache: fixed updating bypassed cached errors (ticket #827).

[Maxim Dounin]

Fix committed, thanks for reporting this.