Opened 7 months ago

Last modified 7 months ago

#2567 new enhancement

sub_filter and gziped payload should trigger warning

Reported by: gl-yziquel@… Owned by:
Priority: minor Milestone:
Component: nginx-core Version: 1.18.x
Keywords: Cc:
uname -a: Linux virtucon 6.2.0-1010-nvidia #10-Ubuntu SMP PREEMPT_DYNAMIC Tue Aug 29 17:57:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.18.0 (Ubuntu)
built with OpenSSL 3.0.2 15 Mar 2022
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -ffile-prefix-map=/build/nginx-zctdR4/nginx-1.18.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/ --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --add-dynamic-module=/build/nginx-zctdR4/nginx-1.18.0/debian/modules/http-geoip2 --with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_sub_module



Reporting an issue that bit me quite some time ago. I don't have the setting to reproduce it because I solved it on my own and moved on. But here is the lesson learned:

I was busy reverse-proxying a website with a path prefix. (The kind of thing you do because you have only one machine, no control over DNS, and want to reuse the same port.) I encountered a Heisenbug because of default headers being passed around by the various tools I was using. But, bottom line:

The reverse proxying didn't work because I had a sub_filter declaration to correct the payloads to account for the prefix in the reverse-proxying. However, the backend gave gziped data. sub_filtering on compressed data is... doomed to failure.

So, for people encountering the same situation, and dealing with the same kind of Heisenbugs: please provide a warning in the logs for people attempting sub_filter on compressed data.

That would have helped me diagnose the problem MUCH MUCH MUCH faster, without having to dissect implicit behaviours related to compression.

Change History (1)

comment:1 by Maxim Dounin, 7 months ago

In theory, the sub_filter is expected to work correctly with arbitrary data, including binary and/or compressed content. As such, I don't think that a runtime warning would be a good solution - or at least there should be a way to quench it.

On the other hand, docs probably should at least mention that compression on the backend might interfere with correct substitutions, and recommend to disable it as needed, either directly on the backend, or by using proxy_set_header Accept-Encoding '';.

Also, it is a long-planned feature in the gunzip module to make it possible to automatically gunzip content based on other modules being used. This looks directly related.

Note: See TracTickets for help on using tickets.