add_after_body concatenates (upstream proxied) gziped content with uncompressed local data

[dwt@…]

I've now spent several days debugging to find out that using Nginx's add_after_body directive, Nginx doesn't decompress and recompress upstream proxied responses when appending the results of the sub request (which yealds uncompressed data).

I'd like to think that this should either produce a warning (as clients tend to discard the uncompressed data appended to the compressed data) or should transparently (maybe optionally) decompress, append and then recompress the http stream to produce output that is actually understandable by clients.

It may also be possible to compress the response from add_after_body and concatenate the compressed response to the compressed upstream servers response (not quite sure about my understanding of gzip here).

In any way, I'd like to suggest to either emit a warning (none was given, even with error_log ... debug;) or form correct (as in readable by clients) responses.

Some excerpts from my config:

include conf.d/error_logging.include;
include conf.d/workers.include;

http {
    # Disabling both gzip and enabling gunzip works around the problem that nginx tries to concatenate 
    # the add_after_body content to gziped upstream responses, which leads to the appended uncompressed
    # content being ignored/discarded by the client.
    # gzip off;
    # gunzip on;
    
    include conf.d/access_logging.include;
    include conf.d/mime_types.include;

    server {
        listen 8000;
        server_name localhost;

        location /proxy_static {
            # returns uncompressed responses
            alias /srv/www/static;
            add_after_body ""; # disable add after body to these locations
        }
        add_after_body /proxy_static/custom_css_and_js.html;
        
        location / {
            # returns gzip compressed responses
            proxy_pass http://host.docker.internal:8001;
        }
    }
    
}

[dwt@…]

I'd like to add that I really appreciated the description of this issue from a fellow developer at ​https://github.com/psf/requests/issues/5887 where he describes this issue as

To be clear, I think the configuration options you have in Nginx are a footgun that Nginx needs to fix. It's too easy to send invalid response bodies that way.

The workaround I found for me is to ensure I do not request gziped content from the upstream proxy server with this:

proxy_set_header Accept-Encoding "";

I would really like to inspire you guys to add some kind of assertion / logging / fix for this to make it possible to at least find this configuration problem more easily and at best to prevent this from becoming an invalid request in the first place.

[dwt@…]

I just found another hard to deal problem with this setup. One of the upstream reverse proxies I have to deal with likes to return cached gziped responses, even when I explicitly request to get _non_ gziped responses. Not yet sure how to work around this one. :-( (Yeah, that's a definitive bug in their config and working with them is the first thing I did - but I have no clue when and if they fix their config).

Of course it would make my life much simpler if nginx could be lenient in what it takes in here and strictly correct in what it returns, i.e. detect upstream returned a gziped response, unzip it, concatenate then and return - maybe even zipped again to cache the gzipped response itself…

Still, dealing with add_after_body isn't quite as much fun as I anticipated.

[Maxim Dounin]

The add_after_body and add_before_body directives simply concatenate the subrequest result with the response body, much like ​SSI include. It's up to the user to ensure concatenated responses are correct, including Content-Encoding used.

Adding an explicit warning if Content-Encoding is used and does not match between subrequests is certainly possible, though probably does not worth the effort: it will noticeably complicate things, as the concat module will have to preserve original Content-Encoding of the main response and inspect Content-Encoding of all subrequests. On the other hand, things won't work anyway, as proper configuration is required for things to work. And the problem is usually immediately visible anyway, while the cases when it is not visible is certainly need to be addressed in clients: it is utterly wrong to silently drop parts of a response.

A simpler solution might be to avoid adding anything to responses with Content-Encoding (or issue a warning when doing so), though the fun fact is: gzipped responses can be concatenated together. The result won't work in major browsers, though technically it is a bug in browsers.

As for the broken servers which always return gzipped responses, you can consider configuring additional proxying through nginx with ​gunzip on; to fix things. Not sure it worth the effort in your setup though.