response vary headers not used in the cache key

[gedl@…]

Scenario:
use nginx to cache server responses based on

• scheme of the request
• host of the proxy
• request uri
• values of the request headers as selected by the response vary headers

Request 1:

curl -H "Authorization: one" -I http://localhost:8081/some_uri
HTTP/1.1 200 OK
Server: nginx
Date: Sat, 11 Nov 2017 01:56:24 GMT
Content-Type: application/json
Content-Length: 1389
Connection: keep-alive
Access-Control-Allow-Headers: Accept, Authorization, Content-Type, x-api-key, Accept-Language
Access-Control-Allow-Methods: GET, POST, PUT, PATCH, OPTIONS, DELETE
Access-Control-Allow-Origin: *
Vary: Authorization
Cache-Control: no-transform, max-age=10, s-maxage=10
request-id: da208a4310e0a67f
loc: local
processing-time: 307
Vary: Accept-Encoding
X-Cached: MISS

Comment: the request was a cache miss. all good

Request 2

 curl -H "Authorization: two" -I http://localhost:8081/api/some_uri
HTTP/1.1 200 OK
Server: nginx
Date: Sat, 11 Nov 2017 01:56:28 GMT
Content-Type: application/json
Content-Length: 1389
Connection: keep-alive
Access-Control-Allow-Headers: Accept, Authorization, Content-Type, x-api-key, Accept-Language
Access-Control-Allow-Methods: GET, POST, PUT, PATCH, OPTIONS, DELETE
Access-Control-Allow-Origin: *
Vary: Authorization
Cache-Control: no-transform, max-age=10, s-maxage=10
request-id: da208a4310e0a67f
loc: local
processing-time: 307
Vary: Accept-Encoding
X-Cached: HIT

Comment: a different value for the authorization header was specified on the second request. As the response includes Vary: Authorization, I'd expect the second request to be a cache MISS. However, it was a cache HIT

Configuration file:

worker_processes 4;


events {
        worker_connections 768;
        multi_accept on;
}

http {

        upstream target {
                server       localhost:8082;
        }

        proxy_cache_path /Users/gluiz/dev/nginx/cache levels=1:2 keys_zone=my_cache:10m max_size=10g inactive=60m use_temp_path=off;
        add_header X-Cached $upstream_cache_status;


server {
        listen       8081;
        server_name  anonymous;
        proxy_intercept_errors off;
        location / {
                 proxy_cache my_cache;
                proxy_set_header Host $http_host;
                proxy_pass http://target$request_uri;
        }

        ssl_verify_client off;
}



        server_tokens off;

}

[gedl@…]

Note: it appears that the problem may be related to the presence of multiple Vary headers in the response. This is RFC accepted, and the semantics is the same as one Vary header with all the values:

Vary: a
Vary: b

is semantically equivalent to:

Vary: a, b

[Maxim Dounin]

Strictly speaking, this not something allowed by RFC. Quoting ​RFC 7230:

   A sender MUST NOT generate multiple header fields with the same field
   name in a message unless either the entire field value for that
   header field is defined as a comma-separated list [i.e., #(values)]
   or the header field is a well-known exception (as noted below).

And the Vary header field is defined as follows, see ​RFC 7231:

     Vary = "*" / 1#field-name

That is, "the entire field value for that header field is defined as a comma-separated list" is not true due to "*", so sender is not allowed to use multiple Vary headers.

On the other hand, the intention is clear enough, and we probably should either respect all the Vary headers, or simply disable caching in such a case.

[gedl@…]

Yes, agreed.

It'd be much more interesting to respect all Vary headers. As a workaround, is there a way of normalising the headers prior to the calculation of the cache key?

[gedl@…]

Hi

Was there any movement on this over the past year?

[Maxim Dounin]

Was there any movement on this over the past year?

I haven't seen any patches submitted to address this.

[martijn.zijlstra@…]

I cant see your used proxy_cache_key so i assume you're using the default :

proxy_cache_key "$scheme$proxy_host$uri$is_args$args";

Changing it to should solve your issue :

proxy_cache_key "$scheme$proxy_host$uri$is_args$args$http_authorization";

[Maxim Dounin <mdounin@…>]

In 8035:cd73509f21e2/nginx:

Upstream: handling of multiple Vary headers (ticket #1423).

Previously, only the last header value was used when caching.

[Maxim Dounin]

Fixed, thanks.