Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#886 closed defect (invalid)

proxy_protocol bug fix in version 1.8.1 broke Nginx for all vhosts

Reported by: jamrok@… Owned by:
Priority: critical Milestone:
Component: nginx-core Version: 1.8.x
Keywords: proxy_protocol Cc:
uname -a: Linux web2 2.6.32-573.8.1.el6.x86_64 #1 SMP Tue Nov 10 18:01:38 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.8.1 built by gcc 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) built with OpenSSL 1.0.1e-fips 11 Feb 2013 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --with-http_spdy_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic'

Description

From nginx 1.8.1 changelog:

*) Bugfix: the "proxy_protocol" parameter of the "listen" directive did

not work if not specified in the first "listen" directive for a
listen socket.

It seems the bug fix above introduced a new bug where specifying the "proxy_protocol" listen directive in any vhost affects ALL vhosts globally instead of only that individual vhost.

The result is that all vhosts (even those without that setting) no longer worked. The automatic system update ran and restarted nginx and all our sites went offline.


Response from Nginx:

# curl -IL localhost
curl: (52) Empty reply from server

Sample Nginx Error Log:

==> /var/log/nginx/error.log <==
" while reading PROXY protocol, client: 10.123.123.123 , server: 0.0.0.0:80
2016/01/27 01:10:38 [error] 1582#0: *269 broken header: "GET /blog/eat-drink-be-merry/ HTTP/1.1
User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
X-Forwarded-For: 157.55.39.99
Accept: */*
Cache-Control: no-cache
X-Forwarded-Proto: http
Host: mysite.com
Pragma: no-cache
X-Cluster-Client-Ip: 157.55.39.99
Connection: Keep-Alive
X-Forwarded-Port: 80
Accept-Encoding: gzip, deflate
From: bingbot(at)microsoft.com


My "fix" was to disable it as shown below:

@@ -2 +2 @@
-        listen 80 proxy_protocol;
+        listen 80;

That "proxy_protocol" option was specified in 4 of dozens of vhosts on a server. It was just "enabled" but not being used in the "real_ip_header" or "proxy_set_header" parameters, so disabling it in those 4 vhosts, allowed for all vhosts to work.

Maybe there was something else that I needed to change, but I don't think that simply enabling that option should have broken all the vhosts.

It was also slightly challenging to find because the vhosts with the setting hadn't been changed for over a year so no "recent" file change with "bad" settings was the cause.

Thanks for looking into it

Change History (1)

comment:1 Changed 4 years ago by arut

  • Resolution set to invalid
  • Status changed from new to closed

After the fix the proxy_protocol setting specified in ANY vhost, will affect ALL vhosts.
Previous behaviour was buggy. Only the first listen directive could enable proxy_protocol
for the vhosts.

When proxy protocol header is read from the socket, it' still unknown which vhost the
request refers to. The host name comes AFTER the proxy protocol header. That means,
we have to make the decision about whether to parse the proxy_protocol header at the point,
where we have no vhost. That decision will obviously be the same for all vhosts.

Looks like your previous proxy_protocol setting had no effect at all.

Last edited 4 years ago by arut (previous) (diff)
Note: See TracTickets for help on using tickets.