Opened 4 months ago

Closed 3 months ago

#1982 closed defect (fixed)

Zero length UDP packets aren't forwarded to upstreams but a lack of response is still counted as an upstream failure

Reported by: d0x2f@… Owned by:
Priority: minor Milestone:
Component: nginx-module Version: 1.18.x
Keywords: udp, stream, proxy Cc:
uname -a: Linux navi 5.6.10-arch1-1 #1 SMP PREEMPT Sat, 02 May 2020 19:11:54 +0000 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.18.0
built with OpenSSL 1.1.1g 21 Apr 2020
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/bin/nginx --pid-path=/run/nginx.pid --lock-path=/run/lock/nginx.lock --user=http --group=http --http-log-path=/var/log/nginx/access.log --error-log-path=stderr --http-client-body-temp-path=/var/lib/nginx/client-body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-cc-opt='-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -D_FORTIFY_SOURCE=2' --with-ld-opt=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now --with-compat --with-debug --with-file-aio --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_geoip_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-pcre-jit --with-stream --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads

Description

Using wireshark I can see that zero length packets aren't forwarded to upstreams, but the nginx logs still report an error after 'fail_timeout'.

2020/05/20 16:46:03 [error] 2046093#2046093: *3 upstream timed out BBB (110: Connection timed out) while proxying connection, udp client: 127.0.0.1, server: 0.0.0.0:1912, upstream: "127.0.0.1:1812", bytes from/to client:0/0, bytes from/to upstream:0/0

More than 'max_fails' upstream errors cause it to be marked as down even though it was never sent any packets.

As a minimal example I have a simple program that just responds to all requests with a static response and with the following nginx config.

If you send a zero length packet (I use sudo nmap -sU 127.0.0.1 -p 1912), it will exhibit the bug.

user nginx;
error_log /var/log/nginx/error.log debug;

events {
  worker_connections 128;
}

stream {
  upstream radius {
    server 127.0.0.1:1812 max_fails=1 fail_timeout=5s;
  }

  server {
    listen 1912 udp;
    proxy_pass radius;
    proxy_timeout 5s;
    proxy_responses 1;
  }
}

http {}

Change History (4)

comment:1 by vl, 4 months ago

Status: newaccepted

Yes, indeed such packets aren't forwarded currently.

Did you hit the issue with some real protocol or it is just manual testing?

comment:2 by d0x2f@…, 4 months ago

The issue came from a production deployment of a radius service.
An empty packet isn't part of the radius protocol as such, but it's occasionally receiving empty packets from web crawlers anyway (specifically internettl.org in this case).

Also of concern is that this exposes the service to a kind of denial of service attack via flooding empty packets, causing nginx to perpetually mark all the upstreams as down.

comment:3 by Vladimir Homutov <vl@…>, 3 months ago

In 7665:d127837c714f/nginx:

Stream: fixed processing of zero length UDP packets (ticket #1982).

comment:4 by vl, 3 months ago

Resolution: fixed
Status: acceptedclosed
Note: See TracTickets for help on using tickets.