Opened 13 days ago
Last modified 4 days ago
#2496 new defect
UDP traffic bandwidth is not limited by proxy_upload_rate and proxy_download_rate
| Reported by: | Owned by: | ||
|---|---|---|---|
| Priority: | minor | Milestone: | |
| Component: | nginx-module | Version: | 1.19.x |
| Keywords: | udp ngx_stream_proxy_module proxy_upload_rate proxy_download_rate | Cc: | m-cieslinski@… |
| uname -a: | Linux host 3.10.0-1160.80.1.el7.x86_64 #1 SMP Tue Nov 8 15:48:59 UTC 2022 x86_64 Linux | ||
| nginx -V: |
nginx version: nginx/1.19.7
built by gcc 10.2.1 20201203 (Alpine 10.2.1_pre1) built with OpenSSL 1.1.1o 3 May 2022 TLS SNI support enabled configure arguments: --with-compat --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --with-perl_modules_path=/usr/lib/perl5/vendor_perl --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-Os' --with-ld-opt=-Wl,--as-needed --with-stream |
||
Description
Hi,
I have problems with bandwidth limit for UDP traffic in Nginx Community Edition (non-enterprise). I've tested TCP/HTTP traffic and bandwidth limit is working fine. But for UDP it is not limiting it at all. Also UDP traffic was tested with DNS config like here https://github.com/tatsushid/nginx-udp-lb-example/blob/master/nginx.conf.template. And still without proper limit applied. Only rate/request per sec is working correctly. I followed this guide https://docs.nginx.com/nginx/admin-guide/security-controls/controlling-access-proxied-tcp/ and https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_upload_rate.
UDP protocol is GELF UDP proxied to upstream Graylog server.
Nginx version tested: 1.19.X and latest.
events {
worker_connections 1024;
}
stream {
upstream gelf_tcp {
hash $remote_addr;
server graylog:12211 fail_timeout=10s;
}
server {
listen 12201;
proxy_timeout 10s;
proxy_download_rate 20;
proxy_upload_rate 11;
proxy_connect_timeout 1s;
proxy_pass gelf_tcp;
}
upstream gelf_udp {
hash $remote_addr;
server graylog:12211 fail_timeout=10s;
}
limit_conn_zone $binary_remote_addr zone=conn_perip:10m;
server {
listen 12201 udp;
proxy_download_rate 20;
proxy_responses 0;
proxy_timeout 1s;
proxy_upload_rate 11;
proxy_pass gelf_udp;
proxy_bind $remote_addr transparent;
# limit_conn conn_perip 5;
# limit_conn_log_level warn;
}
log_format proxy_log '$remote_addr [$time_local] '
'$protocol $status $bytes_sent $bytes_received '
'$session_time "$upstream_addr" '
'"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"';
access_log /var/log/nginx/access.log proxy_log buffer=32k;
error_log /var/log/nginx/error.log warn;
}
Note that with UDP, it is not possible to split a packet into multiple UDP packets. As such,
proxy_upload_rateandproxy_download_raterates are applied on a per-packet basis. That is, as long as nginx hits a limit, it won't try to read any additional packets on the UDP session till the configured rate allows it.For
proxy_download_rate, this means that up to the recv socket buffer (on the socket to the upstream server) and one packet (within nginx itself) can be buffered. Forproxy_upload_rate, since the listening socket is shared among all clients, just one packet (within nginx itself) can be buffered, any additional packets will be dropped.With the mentioned limitations both
proxy_upload_rateandproxy_download_rateeverything works as expected in the tests here. For example, consider the following configuration:stream { server { listen 9000 udp; proxy_pass 127.0.0.1:9001; proxy_download_rate 5; } }And the following test scripts for server and client:
$ cat test-udp-server.pl #!/usr/bin/perl use warnings; use strict; use Socket; use Time::HiRes qw/time/; socket(my $socket, PF_INET, SOCK_DGRAM, 0) or die "socket: $!"; setsockopt($socket, SOL_SOCKET, SO_REUSEPORT, 1) or die "setsockopt: $!"; bind($socket, pack_sockaddr_in(9001, inet_aton("127.0.0.1"))) or die "bind: $!"; my $peer; my $msg = ''; while ($peer = recv($socket, $msg, 65536, 0)) { chomp($msg); my ($port, $ip) = unpack_sockaddr_in($peer); my $s = inet_ntoa($ip) . ':' . $port; my $t = sprintf("%.6f", time()); print "<< ($s $t) $msg\n"; for (1..10) { $t = sprintf("%.6f", time()); print ">> ($s $t) bar\n"; send($socket, "bar\n", 0, $peer); } } $ cat test-udp-client.pl #!/usr/bin/perl use warnings; use strict; use Socket; use Time::HiRes qw/time/; socket(my $socket, PF_INET, SOCK_DGRAM, 0) or die "socket: $!"; setsockopt($socket, SOL_SOCKET, SO_REUSEPORT, 1) or die "setsockopt: $!"; connect($socket, pack_sockaddr_in(9000, inet_aton("127.0.0.1"))) or die "connect: $!"; my $peer; my $msg = ''; my $s = '127.0.0.1:9000'; my $t = sprintf("%.6f", time()); print ">> ($s $t) foo\n"; send($socket, "foo\n", 0); while ($peer = recv($socket, $msg, 65536, 0)) { chomp($msg); my ($port, $ip) = unpack_sockaddr_in($peer); my $s = inet_ntoa($ip) . ':' . $port; my $t = sprintf("%.6f", time()); print "<< ($s $t) $msg\n"; }Test run results on the server side:
Note that 10 response packets are sent almost immediately. And here are the results on the client side:
Note that the first response packet is sent immediately, and other packets are delayed as per
proxy_download_rate.If you still think there is a bug, please provide more details: notably, how do you test, and what do you observe, and what do you expect instead. A simple test script which demonstrates the problem would be awesome.