Opened 5 weeks ago

Closed 5 weeks ago

#2269 closed defect (duplicate)

TLS handshake errors within proxy protocol are reported with incorrect source IP

Reported by: sgielen@… Owned by:
Priority: minor Milestone:
Component: nginx-core Version:
Keywords: Cc: sgielen@…
uname -a: Linux k8s-master2-staging 5.4.0-89-generic #100-Ubuntu SMP Fri Sep 24 14:50:10 UTC 2021 x86_64 Linux
nginx -V: nginx version: nginx/1.20.1
built by gcc 10.2.1 20201203 (Alpine 10.2.1_pre1)
built with OpenSSL 1.1.1k 25 Mar 2021
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx --conf-path=/etc/nginx/nginx.conf --modules-path=/etc/nginx/modules --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-compat --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_geoip_module --with-http_gzip_static_module --with-http_sub_module --with-http_v2_module --with-stream --with-stream_ssl_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-http_secure_link_module --with-http_gunzip_module --with-file-aio --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module --without-http_uwsgi_module --without-http_scgi_module --with-cc-opt='-g -Og -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wno-deprecated-declarations -fno-strict-aliasing -D_FORTIFY_SOURCE=2 --param=ssp-buffer-size=4 -DTCP_FASTOPEN=23 -fPIC -I/root/.hunter/_Base/d45d77d/6f5f962/3b7ee27/Install/include -Wno-cast-function-type -m64 -mtune=native' --with-ld-opt='-fPIE -fPIC -pie -Wl,-z,relro -Wl,-z,now -L/root/.hunter/_Base/d45d77d/6f5f962/3b7ee27/Install/lib' --user=www-data --group=www-data --add-module=/tmp/build/ngx_devel_kit-0.3.1 --add-module=/tmp/build/set-misc-nginx-module-0.32 --add-module=/tmp/build/headers-more-nginx-module-0.33 --add-module=/tmp/build/ngx_http_substitutions_filter_module-b8a71eacc7f986ba091282ab8b1bbbc6ae1807e0 --add-module=/tmp/build/lua-nginx-module-b721656a9127255003b696b42ccc871c7ec18d59 --add-module=/tmp/build/stream-lua-nginx-module-74f8c8bca5b95cecbf42d4e1a465bc08cd075a9b --add-module=/tmp/build/lua-upstream-nginx-module-8aa93ead98ba2060d4efd594ae33a35d153589bf --add-module=/tmp/build/nginx_ajp_module-a964a0bcc6a9f2bfb82a13752d7794a36319ffac --add-dynamic-module=/tmp/build/nginx-http-auth-digest-1.0.0 --add-dynamic-module=/tmp/build/nginx-influxdb-module-5b09391cb7b9a889687c0aa67964c06a2d933e8b --add-dynamic-module=/tmp/build/nginx-opentracing-0.19.0/opentracing --add-dynamic-module=/tmp/build/ModSecurity-nginx-1.0.2 --add-dynamic-module=/tmp/build/ngx_http_geoip2_module-a26c6beed77e81553686852dceb6c7fdacc5970d --add-dynamic-module=/tmp/build/ngx_brotli

Description

I am running nginx as a reverse proxy, receiving connections from a load balancer that uses the proxy protocol. (To be exact, this is the nginx ingress controller running in a Kubernetes cluster receiving connections from the Hetzner Cloud load balancer.)

My access logs show many broken lines like this one:

10.67.1.254 - - [23/Oct/2021:11:17:22 +0000] "x\xA2\xD3\x09\xEFK\xA4\xF3\x0F|!\x03\xC6\xAD\xA5\x85WP\x225\xE6{\xAB\x15\x14\x5C7T\xBCU\xD6\x1BG\xCA\xA4\xDA@\x15\x17\xC3\x8D\xA8\xA1\x84\x9Fq\xC0A \xED\xD8t\x11\xFE\xF6\x8Dm\xF592\x09\xD3\xE9\xDD\xCD\xB3\xCD\xDB\x1A\xD3(\xBDz\x92\xB6\x93\xFF\xED7\x92{\x89x\x0E\x19(\xBC\xBA2P\xAF\x9B\x0B\x9C\x02\xFD/\x17\xC3\xD3\x81[;jc\x8B^\x8A\x99\x7FQ\xC3[\xA8\xD7r\xE1\xB5Q\xDF\xBE\xDFA\x958\xE4F\xA6{\xE6G\xF1\xE1h\xAD:\x0Fv\x0Cy\x96\x80" 400 150 "-" "-" 0 0.125 [] [] - - - - 4dd30d6a031ee9c931798d74445aaaa6

Here, 10.67.1.254 is the IP address of the load balancer, so I initially thought it was sending broken requests. To investigate this, I made a packet capture and found the stream causing the log message above. Its pcap is attached to this ticket.

As you can see in the pcap, the stream correctly starts with a PROXYv1 message with Source Address 93.110.4.143, an IP in Iran unlikely to communicate with this instance for benign reasons. The communication continues with a TLS message that is not decoded by my Wireshark version and does not seem to be a valid TLS handshake.

As such, it seems nginx interprets the message as a plaintext HTTP request and responds with a HTTP/1.1 400 Bad Request.

However, at this point, nginx should have gathered from the Proxy protocol handshake that the source IP is 93.110.4.143 and not 10.67.1.254 as reported in the access logs. I would expect the correct source IP address of the request to be reported, not the IP of the load balancer forwarding it, as intended by the proxy protocol.

(Also, as a second, perhaps lesser issue, I can't find where raw bytes reported in the log message are from. I would expect "F|!" to occur anywhere in the stream, but it does not. Did some decoding take place or does this come from uninitialized memory?)

Attachments (1)

nginx-tcpdump.pcap (2.3 KB ) - added by sgielen@… 5 weeks ago.
PCAP of broken TCP stream between LB and nginx

Download all attachments as: .zip

Change History (2)

by sgielen@…, 5 weeks ago

Attachment: nginx-tcpdump.pcap added

PCAP of broken TCP stream between LB and nginx

comment:1 by Maxim Dounin, 5 weeks ago

Resolution: duplicate
Status: newclosed

Duplicate of #1869.

Note: See TracTickets for help on using tickets.