Lack of "Vary" handling in proxy can lead to corrupted downloads

[Oz Solomon]

Documentation clearly states that the proxy module doesn't handle the Vary response header. However, "not handling" can mean a few things, and currently it means things are broken.

Example:
nginx as reverse proxy to Apache, and Apache is using mod_deflate to compress the content.

User 1 makes a request for content using a modern browser, which nginx passes on to Apache, and Apache happily returns using a compressed response (Content-Encoding: gzip, Vary: Accept-Encoding).

User 2 makes a request for the same content, but using a brain dead browser (in this case IE, which for 0.5% of users does not accept gzip encoding as we've recently measured). Despite the fact that the browser isn't sending "Accept-Encoding: gzip" to the server, the content is served from nginx's cache in compressed format and the browser receives a response it can't handle.

Expected:
nginx should never cache responses that contain the Vary header.

Better Yet, Expected:
nginx should properly handle the Vary header and only serve cached versions if the headers match.

Actual:
Whatever is cached after the first request is served.

Note: For a long form background on this issue, please see: ​http://www.notthewizard.com/2012/02/27/nginx-reverse-proxy-can-cause-ie-to-fail/

[Maxim Dounin]

Yep, Vary handling needs to be implemented. Just disabling cache for responses with the Vary header might be an acceptable solution.

For now, there are two basic workarounds for the particular issue:

1. Use proxy_set_header Accept-Encoding ""; to make sure backend won't return compressed content (or just switch off compression on backend), and do a compression on nginx. (Or vice versa, use proxy_set_header Accept-Encoding "gzip"; and use ​gunzip module to decompress if needed.)

2. Add $http_accept_encoding to the ​cache key.

[Maxim Dounin]

And, just in case, another option which should work too:

proxy_no_cache $upstream_http_vary;

See ​proxy_no_cache.

[Nils Toedtmann]

Hi, i think i ran into with nginx + fastcgi_cache + php-fpm.

Is this the same bug (if it is one)? Observe that the client does not send any "Accept-Encoding:" header:

GET /membership HTTP/1.1
User-Agent: curl/7.21.6 (i686-pc-linux-gnu) libcurl/7.21.6
OpenSSL/1.0.0e zlib/1.2.3.4 libidn/1.22 librtmp/2.3
Host: islington.the-hub.net
Accept: */*

HTTP/1.1 200 OK
Server: nginx/1.4.1
Date: Thu, 16 May 2013 21:48:25 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
X-Powered-By: PHP/5.3.10-1ubuntu3.6
Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT
Content-Encoding: gzip
Vary: Accept-Encoding

Is this bug being worked on?

[Maxim Dounin]

Replying to Nils Toedtmann <nils.toedtmann@gmail.com>:

Is this the same bug (if it is one)? Observe that the client does not send any "Accept-Encoding:" header:

This suggests there is a problem on php side, unlrelated to this bug.

[Sebastian Krebs]

Sorry, that I push this ticket, but is there a chance to get proper "Vary"-handling with for proxy-/fastcgi-caching?

[Maxim Dounin <mdounin@…>]

In 60fde1bc7236691e9218fdfa4c050c1c617aa334/nginx:

Cache: disable caching of responses with Vary (ticket #118).

The "proxy_ignore_header" directive now undersands the "Vary" parameter
to ignore the header as needed.

[Maxim Dounin]

Fixed by 60fde1bc7236 and following commits.