Custom Query (2297 matches)

Will try to describe the problem we have:

We are doing proxy_pass to another domain and change host header (as you can see below), backend returns X-Accel-Redirect with the location of the file to serve, but for some reason on step 3 belowб when nginx receives X-Accel-Redirectб it's using original Host, not the one we changed with proxy_set_header.. which means it's captured by wrong server section and 404 as a result..

curl -I ​http://beta.fastcodesign.com/blablabla/asset_files/-/2014/01/31/telecasts.csv

1) Internally we override Host header to beta.assets.fastcodesign.com and proxy_pass

location ~* ^{/blablabla/(.*) {}

proxy_set_header HOST beta.assets.fastcompany.com; proxy_pass ​http://txjp/$1$is_args$args;

}

upstream txjp {

server localhost:8880;

}

2) Using tcpdump on port 8880 we can see that request goes to backend with new header beta.assets.fastcompany.com

V)..V)..HEAD /asset_files/-/2014/01/31/telecasts.csv HTTP/1.0^{M HOST: beta.assets.fastcompany.com}M Connection: close^{M User-Agent: curl/7.24.0 (x86_64-redhat-linux-gnu) libcurl/7.24.0 NSS/3.12.10.0 zlib/1.2.5 libidn/1.18 libssh2/1.2.2}M Accept: */*^M

HTTP/1.1 200 OK^{M x-rid: 1ok5rjv}M Etag: "165143ec0b37d43cf2c2b7f292647526"^{M Date: Sat, 01 Feb 2014 00:07:47 GMT}M Last-Modified: Fri, 31 Jan 2014 18:15:36 GMT^{M Content-Type: application/json; charset=utf-8}M Content-Length: 1212^{M Cache-Control: max-age=1801}M X-Accel-Redirect: /internal_redirect/assets.fastcompany.com.s3-website-us-east-1.amazonaws.com/asset_files/-/2014/01/31/telecasts.csv^{M X-Response-Time: 69ms}M Connection: close^M

3) X-Accel-Redirect should tell nginx to go fetch that url, but it seems like it uses original host (beta.fastcodesign.com) instead the one we modified with proxy_pass

HEAD /internal_redirect/assets.fastcompany.com.s3-website-us-east-1.amazonaws.com/asset_files/-/2014/01/31/telecasts.csv HTTP/1.0^{M X-Real-IP: 23.23.118.249}M X-Forwarded-For: 23.23.118.249^{M Host: beta.fastcodesign.com}M X-Geo: US^{M Connection: close}M User-Agent: curl/7.24.0 (x86_64-redhat-linux-gnu) libcurl/7.24.0 NSS/3.12.10.0 zlib/1.2.5 libidn/1.18 libssh2/1.2.2^{M Accept: */*}M

TTP/1.1 404 Not Found^{M x-rid: hwii1c}M Content-Type: application/json; charset=utf-8^{M Content-Length: 2}M X-Response-Time: 288ms^{M Date: Sat, 01 Feb 2014 00:07:47 GMT}M Connection: close

Config:

server {
    listen       80;
    server_name  localhost;

    location / {
        proxy_pass http://127.0.0.1:8000/;
    }

    location /int/ {
        internal;
        proxy_pass http://127.0.0.1:8888/;
    }
}

Run 2 upstreams, the first one is for an external location with X-Accel-Redirect:

while true; do printf 'HTTP/1.1 200 OK\r\nX-Accel-Redirect:/int/\r\nCache-Control:no-cache,no-store\r\nContent-Length:0\r\n\r\n'; done | nc -vklp 8000

And the second one is for an internal location:

while true; do printf 'HTTP/1.1 200 OK\r\nContent-Type:text/plain\r\nCache-Control:private,max-age=123\r\nContent-Length:5\r\n\r\nHello'; done | nc -vklp 8888

Run curl to the external location:

$ curl -v 127.0.0.1
*   Trying 127.0.0.1:80...
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.23.3
< Date: Thu, 23 Feb 2023 18:09:40 GMT
< Content-Type: text/plain
< Content-Length: 5
< Connection: keep-alive
< Cache-Control: no-cache,no-store          <--- (!)
< Cache-Control: private,max-age=123        <--- (!)
< 
* Connection #0 to host localhost left intact

Both Cache-Control headers are present in the response. Expected: only the last one.

Some web frameworks add 'Cache-Control:private' by default (sometimes without provided way to change this behaviour). In that case response contains complete duplicates.

It is a very useful feature, as it allows you to override default TTL set for various responses in fastcgi_cache_valid directive, but the problem is that it allows to override all of them, including all 50x errors, so you may end up caching WSOD if the TTL logic used to send X-Accel-Expires header in the upstream app (in our case it is Drupal) has some default TTL with value higher than one second.

It looks like Nginx should allow to override TTL only for those OK default codes: 200, 301 and 302 and never allow to override the TTL set in fastcgi_cache_valid if the response is different than 200, 301 and 302, so all other responses will honor TTL set (if any) in fastcgi_cache_valid instead of accept whatever is send in the X-Accel-Expires header from the backend.