Opened 4 years ago
Closed 4 years ago
#1982 closed defect (fixed)
Zero length UDP packets aren't forwarded to upstreams but a lack of response is still counted as an upstream failure
| Reported by: | Owned by: | ||
|---|---|---|---|
| Priority: | minor | Milestone: | |
| Component: | nginx-module | Version: | 1.18.x |
| Keywords: | udp, stream, proxy | Cc: | |
| uname -a: | Linux navi 5.6.10-arch1-1 #1 SMP PREEMPT Sat, 02 May 2020 19:11:54 +0000 x86_64 GNU/Linux | ||
| nginx -V: |
nginx version: nginx/1.18.0
built with OpenSSL 1.1.1g 21 Apr 2020 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/bin/nginx --pid-path=/run/nginx.pid --lock-path=/run/lock/nginx.lock --user=http --group=http --http-log-path=/var/log/nginx/access.log --error-log-path=stderr --http-client-body-temp-path=/var/lib/nginx/client-body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-cc-opt='-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -D_FORTIFY_SOURCE=2' --with-ld-opt=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now --with-compat --with-debug --with-file-aio --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_geoip_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-pcre-jit --with-stream --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads |
||
Description
Using wireshark I can see that zero length packets aren't forwarded to upstreams, but the nginx logs still report an error after 'fail_timeout'.
2020/05/20 16:46:03 [error] 2046093#2046093: *3 upstream timed out BBB (110: Connection timed out) while proxying connection, udp client: 127.0.0.1, server: 0.0.0.0:1912, upstream: "127.0.0.1:1812", bytes from/to client:0/0, bytes from/to upstream:0/0
More than 'max_fails' upstream errors cause it to be marked as down even though it was never sent any packets.
As a minimal example I have a simple program that just responds to all requests with a static response and with the following nginx config.
If you send a zero length packet (I use sudo nmap -sU 127.0.0.1 -p 1912), it will exhibit the bug.
user nginx;
error_log /var/log/nginx/error.log debug;
events {
worker_connections 128;
}
stream {
upstream radius {
server 127.0.0.1:1812 max_fails=1 fail_timeout=5s;
}
server {
listen 1912 udp;
proxy_pass radius;
proxy_timeout 5s;
proxy_responses 1;
}
}
http {}
Change History (4)
comment:1 by , 4 years ago
| Status: | new → accepted |
|---|
comment:2 by , 4 years ago
The issue came from a production deployment of a radius service.
An empty packet isn't part of the radius protocol as such, but it's occasionally receiving empty packets from web crawlers anyway (specifically internettl.org in this case).
Also of concern is that this exposes the service to a kind of denial of service attack via flooding empty packets, causing nginx to perpetually mark all the upstreams as down.
comment:4 by , 4 years ago
| Resolution: | → fixed |
|---|---|
| Status: | accepted → closed |
Yes, indeed such packets aren't forwarded currently.
Did you hit the issue with some real protocol or it is just manual testing?