Opened 3 years ago

Closed 3 years ago

#2269 closed defect (duplicate)

TLS handshake errors within proxy protocol are reported with incorrect source IP

Reported by: sgielen@… Owned by:
Priority: minor Milestone:
Component: nginx-core Version:
Keywords: Cc: sgielen@…
uname -a: Linux k8s-master2-staging 5.4.0-89-generic #100-Ubuntu SMP Fri Sep 24 14:50:10 UTC 2021 x86_64 Linux
nginx -V: nginx version: nginx/1.20.1
built by gcc 10.2.1 20201203 (Alpine 10.2.1_pre1)
built with OpenSSL 1.1.1k 25 Mar 2021
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx --conf-path=/etc/nginx/nginx.conf --modules-path=/etc/nginx/modules --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/ --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-compat --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_geoip_module --with-http_gzip_static_module --with-http_sub_module --with-http_v2_module --with-stream --with-stream_ssl_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-http_secure_link_module --with-http_gunzip_module --with-file-aio --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module --without-http_uwsgi_module --without-http_scgi_module --with-cc-opt='-g -Og -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wno-deprecated-declarations -fno-strict-aliasing -D_FORTIFY_SOURCE=2 --param=ssp-buffer-size=4 -DTCP_FASTOPEN=23 -fPIC -I/root/.hunter/_Base/d45d77d/6f5f962/3b7ee27/Install/include -Wno-cast-function-type -m64 -mtune=native' --with-ld-opt='-fPIE -fPIC -pie -Wl,-z,relro -Wl,-z,now -L/root/.hunter/_Base/d45d77d/6f5f962/3b7ee27/Install/lib' --user=www-data --group=www-data --add-module=/tmp/build/ngx_devel_kit-0.3.1 --add-module=/tmp/build/set-misc-nginx-module-0.32 --add-module=/tmp/build/headers-more-nginx-module-0.33 --add-module=/tmp/build/ngx_http_substitutions_filter_module-b8a71eacc7f986ba091282ab8b1bbbc6ae1807e0 --add-module=/tmp/build/lua-nginx-module-b721656a9127255003b696b42ccc871c7ec18d59 --add-module=/tmp/build/stream-lua-nginx-module-74f8c8bca5b95cecbf42d4e1a465bc08cd075a9b --add-module=/tmp/build/lua-upstream-nginx-module-8aa93ead98ba2060d4efd594ae33a35d153589bf --add-module=/tmp/build/nginx_ajp_module-a964a0bcc6a9f2bfb82a13752d7794a36319ffac --add-dynamic-module=/tmp/build/nginx-http-auth-digest-1.0.0 --add-dynamic-module=/tmp/build/nginx-influxdb-module-5b09391cb7b9a889687c0aa67964c06a2d933e8b --add-dynamic-module=/tmp/build/nginx-opentracing-0.19.0/opentracing --add-dynamic-module=/tmp/build/ModSecurity-nginx-1.0.2 --add-dynamic-module=/tmp/build/ngx_http_geoip2_module-a26c6beed77e81553686852dceb6c7fdacc5970d --add-dynamic-module=/tmp/build/ngx_brotli


I am running nginx as a reverse proxy, receiving connections from a load balancer that uses the proxy protocol. (To be exact, this is the nginx ingress controller running in a Kubernetes cluster receiving connections from the Hetzner Cloud load balancer.)

My access logs show many broken lines like this one: - - [23/Oct/2021:11:17:22 +0000] "x\xA2\xD3\x09\xEFK\xA4\xF3\x0F|!\x03\xC6\xAD\xA5\x85WP\x225\xE6{\xAB\x15\x14\x5C7T\xBCU\xD6\x1BG\xCA\xA4\xDA@\x15\x17\xC3\x8D\xA8\xA1\x84\x9Fq\xC0A \xED\xD8t\x11\xFE\xF6\x8Dm\xF592\x09\xD3\xE9\xDD\xCD\xB3\xCD\xDB\x1A\xD3(\xBDz\x92\xB6\x93\xFF\xED7\x92{\x89x\x0E\x19(\xBC\xBA2P\xAF\x9B\x0B\x9C\x02\xFD/\x17\xC3\xD3\x81[;jc\x8B^\x8A\x99\x7FQ\xC3[\xA8\xD7r\xE1\xB5Q\xDF\xBE\xDFA\x958\xE4F\xA6{\xE6G\xF1\xE1h\xAD:\x0Fv\x0Cy\x96\x80" 400 150 "-" "-" 0 0.125 [] [] - - - - 4dd30d6a031ee9c931798d74445aaaa6

Here, is the IP address of the load balancer, so I initially thought it was sending broken requests. To investigate this, I made a packet capture and found the stream causing the log message above. Its pcap is attached to this ticket.

As you can see in the pcap, the stream correctly starts with a PROXYv1 message with Source Address, an IP in Iran unlikely to communicate with this instance for benign reasons. The communication continues with a TLS message that is not decoded by my Wireshark version and does not seem to be a valid TLS handshake.

As such, it seems nginx interprets the message as a plaintext HTTP request and responds with a HTTP/1.1 400 Bad Request.

However, at this point, nginx should have gathered from the Proxy protocol handshake that the source IP is and not as reported in the access logs. I would expect the correct source IP address of the request to be reported, not the IP of the load balancer forwarding it, as intended by the proxy protocol.

(Also, as a second, perhaps lesser issue, I can't find where raw bytes reported in the log message are from. I would expect "F|!" to occur anywhere in the stream, but it does not. Did some decoding take place or does this come from uninitialized memory?)

Attachments (1)

nginx-tcpdump.pcap (2.3 KB ) - added by sgielen@… 3 years ago.
PCAP of broken TCP stream between LB and nginx

Download all attachments as: .zip

Change History (2)

by sgielen@…, 3 years ago

Attachment: nginx-tcpdump.pcap added

PCAP of broken TCP stream between LB and nginx

comment:1 by Maxim Dounin, 3 years ago

Resolution: duplicate
Status: newclosed

Duplicate of #1869.

Note: See TracTickets for help on using tickets.