Opened 3 years ago
Closed 3 years ago
#2269 closed defect (duplicate)
TLS handshake errors within proxy protocol are reported with incorrect source IP
| Reported by: | Owned by: | ||
|---|---|---|---|
| Priority: | minor | Milestone: | |
| Component: | nginx-core | Version: | |
| Keywords: | Cc: | sgielen@… | |
| uname -a: | Linux k8s-master2-staging 5.4.0-89-generic #100-Ubuntu SMP Fri Sep 24 14:50:10 UTC 2021 x86_64 Linux | ||
| nginx -V: |
nginx version: nginx/1.20.1
built by gcc 10.2.1 20201203 (Alpine 10.2.1_pre1) built with OpenSSL 1.1.1k 25 Mar 2021 TLS SNI support enabled configure arguments: --prefix=/usr/local/nginx --conf-path=/etc/nginx/nginx.conf --modules-path=/etc/nginx/modules --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-compat --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_geoip_module --with-http_gzip_static_module --with-http_sub_module --with-http_v2_module --with-stream --with-stream_ssl_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-http_secure_link_module --with-http_gunzip_module --with-file-aio --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module --without-http_uwsgi_module --without-http_scgi_module --with-cc-opt='-g -Og -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wno-deprecated-declarations -fno-strict-aliasing -D_FORTIFY_SOURCE=2 --param=ssp-buffer-size=4 -DTCP_FASTOPEN=23 -fPIC -I/root/.hunter/_Base/d45d77d/6f5f962/3b7ee27/Install/include -Wno-cast-function-type -m64 -mtune=native' --with-ld-opt='-fPIE -fPIC -pie -Wl,-z,relro -Wl,-z,now -L/root/.hunter/_Base/d45d77d/6f5f962/3b7ee27/Install/lib' --user=www-data --group=www-data --add-module=/tmp/build/ngx_devel_kit-0.3.1 --add-module=/tmp/build/set-misc-nginx-module-0.32 --add-module=/tmp/build/headers-more-nginx-module-0.33 --add-module=/tmp/build/ngx_http_substitutions_filter_module-b8a71eacc7f986ba091282ab8b1bbbc6ae1807e0 --add-module=/tmp/build/lua-nginx-module-b721656a9127255003b696b42ccc871c7ec18d59 --add-module=/tmp/build/stream-lua-nginx-module-74f8c8bca5b95cecbf42d4e1a465bc08cd075a9b --add-module=/tmp/build/lua-upstream-nginx-module-8aa93ead98ba2060d4efd594ae33a35d153589bf --add-module=/tmp/build/nginx_ajp_module-a964a0bcc6a9f2bfb82a13752d7794a36319ffac --add-dynamic-module=/tmp/build/nginx-http-auth-digest-1.0.0 --add-dynamic-module=/tmp/build/nginx-influxdb-module-5b09391cb7b9a889687c0aa67964c06a2d933e8b --add-dynamic-module=/tmp/build/nginx-opentracing-0.19.0/opentracing --add-dynamic-module=/tmp/build/ModSecurity-nginx-1.0.2 --add-dynamic-module=/tmp/build/ngx_http_geoip2_module-a26c6beed77e81553686852dceb6c7fdacc5970d --add-dynamic-module=/tmp/build/ngx_brotli |
||
Description
I am running nginx as a reverse proxy, receiving connections from a load balancer that uses the proxy protocol. (To be exact, this is the nginx ingress controller running in a Kubernetes cluster receiving connections from the Hetzner Cloud load balancer.)
My access logs show many broken lines like this one:
10.67.1.254 - - [23/Oct/2021:11:17:22 +0000] "x\xA2\xD3\x09\xEFK\xA4\xF3\x0F|!\x03\xC6\xAD\xA5\x85WP\x225\xE6{\xAB\x15\x14\x5C7T\xBCU\xD6\x1BG\xCA\xA4\xDA@\x15\x17\xC3\x8D\xA8\xA1\x84\x9Fq\xC0A \xED\xD8t\x11\xFE\xF6\x8Dm\xF592\x09\xD3\xE9\xDD\xCD\xB3\xCD\xDB\x1A\xD3(\xBDz\x92\xB6\x93\xFF\xED7\x92{\x89x\x0E\x19(\xBC\xBA2P\xAF\x9B\x0B\x9C\x02\xFD/\x17\xC3\xD3\x81[;jc\x8B^\x8A\x99\x7FQ\xC3[\xA8\xD7r\xE1\xB5Q\xDF\xBE\xDFA\x958\xE4F\xA6{\xE6G\xF1\xE1h\xAD:\x0Fv\x0Cy\x96\x80" 400 150 "-" "-" 0 0.125 [] [] - - - - 4dd30d6a031ee9c931798d74445aaaa6
Here, 10.67.1.254 is the IP address of the load balancer, so I initially thought it was sending broken requests. To investigate this, I made a packet capture and found the stream causing the log message above. Its pcap is attached to this ticket.
As you can see in the pcap, the stream correctly starts with a PROXYv1 message with Source Address 93.110.4.143, an IP in Iran unlikely to communicate with this instance for benign reasons. The communication continues with a TLS message that is not decoded by my Wireshark version and does not seem to be a valid TLS handshake.
As such, it seems nginx interprets the message as a plaintext HTTP request and responds with a HTTP/1.1 400 Bad Request.
However, at this point, nginx should have gathered from the Proxy protocol handshake that the source IP is 93.110.4.143 and not 10.67.1.254 as reported in the access logs. I would expect the correct source IP address of the request to be reported, not the IP of the load balancer forwarding it, as intended by the proxy protocol.
(Also, as a second, perhaps lesser issue, I can't find where raw bytes reported in the log message are from. I would expect "F|!" to occur anywhere in the stream, but it does not. Did some decoding take place or does this come from uninitialized memory?)
PCAP of broken TCP stream between LB and nginx