Opened 9 years ago

Closed 9 years ago

Last modified 9 years ago

#858 closed defect (fixed)

Use of PROXY protocol return HTTP 400 error

Reported by: Jonathan Leroy Owned by:
Priority: minor Milestone:
Component: nginx-core Version: 1.8.x
Keywords: Cc: jonathan@…
uname -a: Linux b2o-web3-l 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.8.0
built with OpenSSL 1.0.1k 8 Jan 2015
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2' --with-ld-opt=-Wl,-z,relro --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_gunzip_module --with-file-aio --with-threads --with-http_spdy_module --with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_secure_link_module --with-http_sub_module --with-http_xslt_module --with-mail --with-mail_ssl_module --add-module=/usr/src/builddir/debian/modules/nginx-auth-pam --add-module=/usr/src/builddir/debian/modules/nginx-dav-ext-module --add-module=/usr/src/builddir/debian/modules/nginx-echo --add-module=/usr/src/builddir/debian/modules/nginx-upstream-fair --add-module=/usr/src/builddir/debian/modules/ngx_http_substitutions_filter_module --add-module=/usr/src/builddir/debian/modules/nginx-cache-purge --add-module=/usr/src/builddir/debian/modules/ngx_http_pinba_module --add-module=/usr/src/builddir/debian/modules/nginx-x-rid-header --with-ld-opt=-lossp-uuid

Description

Nginx return an HTTP 400 (Bad Request) error when I configure HAProxy to use PROXY protocol while communicate with it.

Nginx doesn't seems to recognize the PROXY header.

My Nginx config :

set_real_ip_from fd41:9eff:0c33::/48;
real_ip_header proxy_protocol;

server {
    listen [fd41:9eff:c33::3]:443;

    server_name www.domain.tld;
    root /var/www/www.domain.tld/public;
 
    access_log syslog:server=[fd41:9eff:c33::1]:514,facility=local7,severity=info,tag=nginx_access combined;
    error_log  syslog:server=[fd41:9eff:c33::1]:514,facility=local7,severity=info,tag=nginx_errors;

    # HTTP Strict Transport Security (HSTS). 1 year.
    add_header Strict-Transport-Security max-age=31536000;
    
    expires $expires;
    client_max_body_size 25m;
    
    index index.php index.html;
    
    location / {
        try_files $uri $uri/ /index.php$is_args$args;
    }

    location ~ ^(.+\.php)(.*)$ {
        try_files $uri =403;
        
        #fastcgi_buffers 8 16k;
        #fastcgi_buffer_size 32k;
        fastcgi_read_timeout 180s;
        fastcgi_split_path_info ^(.+\.php)(.*)$;
        
        include fastcgi.conf;
        fastcgi_param SERVER_NAME $http_host;
        fastcgi_param PATH_INFO $fastcgi_path_info;
        fastcgi_param PATH_TRANSLATED $document_root$fastcgi_path_info;
        fastcgi_param HTTPS $https;
        #fastcgi_param PHP_VALUE "sendmail_path=/usr/sbin/sendmail -t -i -f $http_host@domain.tld";
        fastcgi_pass unix:/var/run/php5-fpm-www.domain.tld.sock;
    }
    
    location ~ /\. {
        deny all;
        access_log off;
        log_not_found off;
    }
    
    location = /favicon.ico {
      access_log off;
      log_not_found off;
    }
}

server {
    listen [fd41:9eff:c33::3]:443 default_server ssl spdy proxy_protocol;

    server_name domain.tld;

    access_log off;
    error_log /dev/null;
    
    # HTTP Strict Transport Security (HSTS). 1 year.
    add_header Strict-Transport-Security max-age=31536000;
    
    ssl_certificate /etc/XXXXXX/ssl/certs/domain.tld-wildcard/domain.tld-wildcard+dvcasha2.crt;
    ssl_certificate_key /etc/XXXXXX/ssl/certs/domain.tld-wildcard/domain.tld-wildcard.key;
    #ssl_stapling on;
    #ssl_stapling_verify on;
    #ssl_trusted_certificate /etc/XXXXXX/ssl/CA/Certum/OCSP/CTNCA+dvcasha2.pem;

    return 301 https://www.domain.tld$request_uri;
}

server {
    listen [fd41:9eff:c33::3]:80 default_server proxy_protocol;

    server_name domain.tld www.domain.tld;

    access_log off;
    error_log /dev/null;

    return 301 https://www.domain.tld$request_uri;
}

Nginx error log :

Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 accept: [fd41:9eff:c33::ffff]:36222 fd:44
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 event timer add: 44: 60000:1449845703579
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 reusable connection: 1
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 epoll add event: fd:44 op:1 ev:80002001
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 post event 00000000018E5748
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 delete posted event 00000000018E5748
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http wait request handler
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 malloc: 000000000194ACD0:1024
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 recv: fd:44 258 of 1024
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 reusable connection: 0
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 posix_memalign: 0000000001924B90:4096 @16
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http process request line
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [info] 31660#31660: *3055 client sent invalid request while reading client request line, client: fd41:9eff:c33::ffff, server: collectorsquare.com, request: "PROXY TCP4 178.250.6.44 178.33.237.85 52692 80"
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http finalize request: 400, "?" a:1, c:1
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 event timer del: 44: 1449845703579
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http special response: 400, "?"
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http set discard body
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 xslt filter header
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 HTTP/1.1 400 Bad Request#015#012Server: nginx/1.8.0#015#012Date: Fri, 11 Dec 2015 14:54:03 GMT#015#012Content-Type: text/html#015#012Content-Length: 172#015#012Connection: close#015
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 write new buf t:1 f:0 00000000019259A0, pos 00000000019259A0, size: 151 file: 0, size: 0
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http write filter: l:0 f:0 s:151
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http output filter "?"
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http copy filter: "?"
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 posix_memalign: 0000000001925BA0:4096 @16
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 image filter
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 xslt filter body
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http postpone filter "?" 0000000001925B78
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 write old buf t:1 f:0 00000000019259A0, pos 00000000019259A0, size: 151 file: 0, size: 0
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 write new buf t:0 f:0 0000000000000000, pos 00000000006FAF40, size: 120 file: 0, size: 0
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 write new buf t:0 f:0 0000000000000000, pos 00000000006FB460, size: 52 file: 0, size: 0
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http write filter: l:1 f:0 s:323
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http write filter limit 0
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 writev: 323 of 323
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http write filter 0000000000000000
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http copy filter: 0 "?"
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http finalize request: 0, "?" a:1, c:1
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 event timer add: 44: 5000:1449845648579
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http lingering close handler
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 recv: fd:44 -1 of 4096
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 recv() not ready (11: Resource temporarily unavailable)
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 lingering read: -2
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 event timer: 44, old: 1449845648579, new: 1449845648579
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 post event 00000000018E5748
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 delete posted event 00000000018E5748
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http lingering close handler
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 recv: fd:44 0 of 4096
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 lingering read: 0
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http request count:1 blk:0
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http close request
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http log handler
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 http pinba handler
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 free: 0000000001924B90, unused: 8
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 free: 0000000001925BA0, unused: 4016
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 close http connection: 44
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 event timer del: 44: 1449845648579
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 reusable connection: 0
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 free: 000000000194ACD0
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 free: 0000000001910730, unused: 0
Dec 11 15:54:03 server3 nginx_errors: 2015/12/11 15:54:03 [debug] 31660#31660: *3055 free: 0000000001910490, unused: 48

Change History (8)

comment:1 by Maxim Dounin, 9 years ago

Resolution: fixed
Status: newclosed

Likely you are facing the problem fixed in nginx 1.9.0:

    *) Bugfix: the "proxy_protocol" parameter of the "listen" directive did
       not work if not specified in the first "listen" directive for a
       listen socket.

When using older versions, including 1.8.0, make sure to enable proxy_protocol in the first listen directive for a given listening socket (or all of them).

comment:2 by Jonathan Leroy, 9 years ago

Resolution: fixed
Status: closedreopened

Hi Maxim,

I've tried to enable proxy_protocol directive on all listen directives, but Nginx still return the same error.

comment:3 by Maxim Dounin, 9 years ago

Make sure you've really modified all listen directives, not just directives in a particular file. Note that nginx configuration is in nginx.conf, and everything else are just included files. Please also make sure that you've reloaded nginx configuration after changing it - and make sure to check logs after configuration reload, as nginx may not be able switch to new configuration if there are errors.

If you still think you see a new bug, please make sure to:

  • test if you are able to reproduce it using latest nginx (1.9.9 currently) without any 3rd party modules/patches;
  • provide full minimal nginx configuration you are able to reproduce the problem with (nginx.conf and all included files).

comment:4 by Jonathan Leroy, 9 years ago

I've upgraded one of my backend servers to Nginx 1.9.9 and this issue persists :

Dec 11 18:34:41 server3 nginx_errors: 2015/12/11 18:34:41 [info] 3965#3965: *390 client sent invalid request while reading client request line, client: fd41:9eff:c33::ffff, server: collectorsquare.com, request: "PROXY TCP4 157.55.39.2 178.33.237.85 65360 80"
Dec 11 18:34:42 server3 nginx_errors: 2015/12/11 18:34:42 [info] 3965#3965: *395 client sent invalid request while reading client request line, client: fd41:9eff:c33::ffff, server: collectorsquare.com, request: "PROXY TCP6 2a01:e34:ec51:42b0:80ec:6adb:8d18:e235 2001:41d0:8:2e55::1 56553 80"

I've tried a minimal Nginx config, without success :

#user  nginx;
user www-data;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    #include /etc/nginx/conf.d/*.conf;
    #include /etc/nginx/sites-enabled/*;

#set_real_ip_from fd41:9eff:0c33::/48;
#real_ip_header proxy_protocol;

server {
    listen [fd41:9eff:c33::3]:80 proxy_protocol;

    server_name domain.tld;

    access_log off;
    error_log /dev/null;

    return 301 https://www.domain.tld$request_uri;
}

}

comment:5 by Maxim Dounin, 9 years ago

Status: reopenedaccepted

Thanks, I was able to reproduce the problem here. It looks like an issue with IPv6 listening sockets. Please try the following patch:

# HG changeset patch
# User Maxim Dounin <mdounin@mdounin.ru>
# Date 1450107697 -10800
#      Mon Dec 14 18:41:37 2015 +0300
# Node ID 7149c07976dc71f9d74e8b7ad99e6607a004c57f
# Parent  0e8a8155b560e5fb1e8117bdec75e903d618fa0c
Fixed PROXY protocol on IPv6 sockets (ticket #858).

diff --git a/src/http/ngx_http.c b/src/http/ngx_http.c
--- a/src/http/ngx_http.c
+++ b/src/http/ngx_http.c
@@ -1920,6 +1920,7 @@ ngx_http_add_addrs6(ngx_conf_t *cf, ngx_
 #if (NGX_HTTP_V2)
         addrs6[i].conf.http2 = addr[i].opt.http2;
 #endif
+        addrs6[i].conf.proxy_protocol = addr[i].opt.proxy_protocol;
 
         if (addr[i].hash.buckets == NULL
             && (addr[i].wc_head == NULL

comment:6 by Maxim Dounin <mdounin@…>, 9 years ago

In 6331:ceeb1edb3018/nginx:

Fixed PROXY protocol on IPv6 sockets (ticket #858).

comment:7 by Maxim Dounin, 9 years ago

Resolution: fixed
Status: acceptedclosed

Fix committed.

comment:8 by Jonathan Leroy, 9 years ago

Sorry for my late reply.

After applying your patch, proxy_protocol directive now works. I've checked with HTTP, HTTPS, and HTTPS+HTTP2.
Thank you !

Last edited 9 years ago by Jonathan Leroy (previous) (diff)
Note: See TracTickets for help on using tickets.