#886 closed defect (invalid)
proxy_protocol bug fix in version 1.8.1 broke Nginx for all vhosts
| Reported by: | Owned by: | ||
|---|---|---|---|
| Priority: | critical | Milestone: | |
| Component: | nginx-core | Version: | 1.8.x |
| Keywords: | proxy_protocol | Cc: | |
| uname -a: | Linux web2 2.6.32-573.8.1.el6.x86_64 #1 SMP Tue Nov 10 18:01:38 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux | ||
| nginx -V: |
nginx version: nginx/1.8.1
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) built with OpenSSL 1.0.1e-fips 11 Feb 2013 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --with-http_spdy_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' |
||
Description
From nginx 1.8.1 changelog:
*) Bugfix: the "proxy_protocol" parameter of the "listen" directive did
not work if not specified in the first "listen" directive for a
listen socket.
It seems the bug fix above introduced a new bug where specifying the "proxy_protocol" listen directive in any vhost affects ALL vhosts globally instead of only that individual vhost.
The result is that all vhosts (even those without that setting) no longer worked. The automatic system update ran and restarted nginx and all our sites went offline.
Response from Nginx:
# curl -IL localhost curl: (52) Empty reply from server
Sample Nginx Error Log:
==> /var/log/nginx/error.log <== " while reading PROXY protocol, client: 10.123.123.123 , server: 0.0.0.0:80 2016/01/27 01:10:38 [error] 1582#0: *269 broken header: "GET /blog/eat-drink-be-merry/ HTTP/1.1 User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) X-Forwarded-For: 157.55.39.99 Accept: */* Cache-Control: no-cache X-Forwarded-Proto: http Host: mysite.com Pragma: no-cache X-Cluster-Client-Ip: 157.55.39.99 Connection: Keep-Alive X-Forwarded-Port: 80 Accept-Encoding: gzip, deflate From: bingbot(at)microsoft.com
My "fix" was to disable it as shown below:
@@ -2 +2 @@ - listen 80 proxy_protocol; + listen 80;
That "proxy_protocol" option was specified in 4 of dozens of vhosts on a server. It was just "enabled" but not being used in the "real_ip_header" or "proxy_set_header" parameters, so disabling it in those 4 vhosts, allowed for all vhosts to work.
Maybe there was something else that I needed to change, but I don't think that simply enabling that option should have broken all the vhosts.
It was also slightly challenging to find because the vhosts with the setting hadn't been changed for over a year so no "recent" file change with "bad" settings was the cause.
Thanks for looking into it
After the fix the proxy_protocol setting specified in ANY vhost, will affect ALL vhosts.
Previous behaviour was buggy. Only the first listen directive could enable proxy_protocol
for the vhosts.
When proxy protocol header is read from the socket, it' still unknown which vhost the
request refers to. The host name comes AFTER the proxy protocol header. That means,
we have to make the decision about whether to parse the proxy_protocol header at the point,
where we have no vhost. That decision will obviously be the same for all vhosts.
Looks like your previous proxy_protocol setting had no effect at all.