Opened 2 years ago

Closed 19 months ago

#827 closed defect (fixed)

proxy_cache bug when using custom error pages

Reported by: stackoverflow.com/users/220086/denis-pshenov Owned by: mdounin
Priority: major Milestone:
Component: nginx-module Version: 1.8.x
Keywords: proxy_cache ngx_http_proxy_module Cc:
Sensitive: no
uname -a: Linux footyroom.next 3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u2 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.8.0 built with OpenSSL 1.0.1e 11 Feb 2013 TLS SNI support enabled configure arguments: --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2' --with-ld-opt=-Wl,-z,relro --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_gunzip_module --with-file-aio --with-threads --with-http_spdy_module --with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_secure_link_module --with-http_sub_module --with-http_xslt_module --with-mail --with-mail_ssl_module --add-module=/usr/src/builddir/debian/modules/nginx-auth-pam --add-module=/usr/src/builddir/debian/modules/nginx-dav-ext-module --add-module=/usr/src/builddir/debian/modules/nginx-echo --add-module=/usr/src/builddir/debian/modules/nginx-upstream-fair --add-module=/usr/src/builddir/debian/modules/ngx_http_substitutions_filter_module --add-module=/usr/src/builddir/debian/modules/nginx-cache-purge --add-module=/usr/src/builddir/debian/modules/ngx_http_pinba_module --add-module=/usr/src/builddir/debian/modules/nginx-x-rid-header --with-ld-opt=-lossp-uuid

Description

Hi,

I have a very simple proxy config:

http {

proxy_cache_path /var/www/cache levels=1:2 keys_zone=s3-images-cache:50m inactive=1M max_size=1000m;
proxy_temp_path /var/www/cache/tmp;

server {

listen 80;
server_name images.example.com;

location / {

proxy_cache s3-images-cache;
proxy_cache_key $scheme$proxy_host$uri$is_args$args;
proxy_cache_bypass $http_purge_cache;
proxy_cache_valid any 1y;
proxy_pass http://images-example.s3.amazonaws.com;
add_header X-Cache $upstream_cache_status;

proxy_intercept_errors on;
error_page 404 = @no_image;

}

location @no_image {

return 403;

}

}

}

Now follow me here:

  1. Let's request /image.jpg.
  2. Request is sent to proxy for /image.jpg (does not exists yet).
  3. Backend responds with 404.
  4. "proxy_intercept_errors on" kicks in and "error_page 404 = @no_image" is called.
  5. Nginx returns 403.
  6. Do another request for same image and see that "X-Cache: HIT" is set. We are clearly hitting cache.

But, if we check /var/www/cache/ folder at this time we will see that there is no cache item created for this request. So does it mean Nginx keeps the cache for it in memory and forgot to write to file?

  1. Let's upload /image.jpg to backend.
  2. Now do "PURGE-CACHE: 1" request to that image. We see that now we get the image instead of 403. Good.

If we check /var/www/cache/ we will see that the cache file is now finally created for this request. Also good.

  1. Now here is the problem: lets request /image.jpg again.
  2. Nginx returns 403 with "X-Cache: HIT". Why? So it's hitting the cache but returning something else not what is in /var/www/cache folder?? How?

My only explanation to this is it seems that Nginx is caching the response in memory and doesn't write to file when we are hitting error with our custom error_page in the proxied responses. Furthermore when using proxy_cache_bypass it does not overwrite in-memory cache, so that subsequent requests to the same item will be using old cache which is stored in memory and not the new one created in the cache folder.

Change History (3)

comment:1 Changed 19 months ago by mdounin

  • Owner set to mdounin
  • Status changed from new to assigned

When caching errors which are intercepted using proxy_intercept_errors, nginx only stores error code in memory (and nothing more). That's why you see cache used but no cache file at step 6.

As for the errors returned again at step 10, it looks like a bug related to proxy_cache_bypass. The following patch should fix it:

# HG changeset patch
# User Maxim Dounin <mdounin@mdounin.ru>
# Date 1463067394 -10800
#      Thu May 12 18:36:34 2016 +0300
# Node ID 26a9444d07f49cd8b3c0fbc4e167999b4ef45f8e
# Parent  3aed80d49dd8eb32482ce2d277ddd2b3d77b7363
Cache: fixed updating bypassed cached errors (ticket #827).

diff --git a/src/http/ngx_http_file_cache.c b/src/http/ngx_http_file_cache.c
--- a/src/http/ngx_http_file_cache.c
+++ b/src/http/ngx_http_file_cache.c
@@ -1403,6 +1403,7 @@ ngx_http_file_cache_update(ngx_http_requ
     ngx_shmtx_lock(&cache->shpool->mutex);
 
     c->node->count--;
+    c->node->error = 0;
     c->node->uniq = uniq;
     c->node->body_start = c->body_start;
 

comment:2 Changed 19 months ago by Maxim Dounin <mdounin@…>

In 6539:d94f57990492/nginx:

Cache: fixed updating bypassed cached errors (ticket #827).

comment:3 Changed 19 months ago by mdounin

  • Resolution set to fixed
  • Status changed from assigned to closed

Fix committed, thanks for reporting this.

Note: See TracTickets for help on using tickets.