Opened 13 days ago

Last modified 4 days ago

#2496 new defect

UDP traffic bandwidth is not limited by proxy_upload_rate and proxy_download_rate

Reported by: m-cieslinski@… Owned by:
Priority: minor Milestone:
Component: nginx-module Version: 1.19.x
Keywords: udp ngx_stream_proxy_module proxy_upload_rate proxy_download_rate Cc: m-cieslinski@…
uname -a: Linux host 3.10.0-1160.80.1.el7.x86_64 #1 SMP Tue Nov 8 15:48:59 UTC 2022 x86_64 Linux
nginx -V: nginx version: nginx/1.19.7
built by gcc 10.2.1 20201203 (Alpine 10.2.1_pre1)
built with OpenSSL 1.1.1o 3 May 2022
TLS SNI support enabled
configure arguments: --with-compat --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --with-perl_modules_path=/usr/lib/perl5/vendor_perl --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-Os' --with-ld-opt=-Wl,--as-needed --with-stream

Description

Hi,
I have problems with bandwidth limit for UDP traffic in Nginx Community Edition (non-enterprise). I've tested TCP/HTTP traffic and bandwidth limit is working fine. But for UDP it is not limiting it at all. Also UDP traffic was tested with DNS config like here https://github.com/tatsushid/nginx-udp-lb-example/blob/master/nginx.conf.template. And still without proper limit applied. Only rate/request per sec is working correctly. I followed this guide https://docs.nginx.com/nginx/admin-guide/security-controls/controlling-access-proxied-tcp/ and https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_upload_rate.
UDP protocol is GELF UDP proxied to upstream Graylog server.
Nginx version tested: 1.19.X and latest.

events {
worker_connections 1024;

}

stream {

upstream gelf_tcp {

hash $remote_addr;
server graylog:12211 fail_timeout=10s;

}

server {

listen 12201;
proxy_timeout 10s;
proxy_download_rate 20;
proxy_upload_rate 11;
proxy_connect_timeout 1s;
proxy_pass gelf_tcp;

}

upstream gelf_udp {

hash $remote_addr;
server graylog:12211 fail_timeout=10s;

}
limit_conn_zone $binary_remote_addr zone=conn_perip:10m;

server {

listen 12201 udp;
proxy_download_rate 20;
proxy_responses 0;
proxy_timeout 1s;
proxy_upload_rate 11;
proxy_pass gelf_udp;
proxy_bind $remote_addr transparent;
# limit_conn conn_perip 5;
# limit_conn_log_level warn;

}
log_format proxy_log '$remote_addr [$time_local] '

'$protocol $status $bytes_sent $bytes_received '
'$session_time "$upstream_addr" '
'"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"';

access_log /var/log/nginx/access.log proxy_log buffer=32k;
error_log /var/log/nginx/error.log warn;

}

Change History (1)

comment:1 by Maxim Dounin, 4 days ago

Only rate/request per sec is working correctly.

Note that with UDP, it is not possible to split a packet into multiple UDP packets. As such, proxy_upload_rate and proxy_download_rate rates are applied on a per-packet basis. That is, as long as nginx hits a limit, it won't try to read any additional packets on the UDP session till the configured rate allows it.

For proxy_download_rate, this means that up to the recv socket buffer (on the socket to the upstream server) and one packet (within nginx itself) can be buffered. For proxy_upload_rate, since the listening socket is shared among all clients, just one packet (within nginx itself) can be buffered, any additional packets will be dropped.

With the mentioned limitations both proxy_upload_rate and proxy_download_rate everything works as expected in the tests here. For example, consider the following configuration:

stream {
    server {
        listen 9000 udp;
        proxy_pass 127.0.0.1:9001;
        proxy_download_rate 5;
    }
}

And the following test scripts for server and client:

$ cat test-udp-server.pl 
#!/usr/bin/perl

use warnings;
use strict;

use Socket;
use Time::HiRes qw/time/;

socket(my $socket, PF_INET, SOCK_DGRAM, 0)
	or die "socket: $!";
setsockopt($socket, SOL_SOCKET, SO_REUSEPORT, 1)
	or die "setsockopt: $!";
bind($socket, pack_sockaddr_in(9001, inet_aton("127.0.0.1")))
	or die "bind: $!";

my $peer;
my $msg = '';

while ($peer = recv($socket, $msg, 65536, 0)) {

	chomp($msg);
	my ($port, $ip) = unpack_sockaddr_in($peer);
	my $s = inet_ntoa($ip) . ':' . $port;
        my $t = sprintf("%.6f", time());

	print "<< ($s $t) $msg\n";

	for (1..10) {
		$t = sprintf("%.6f", time());
		print ">> ($s $t) bar\n";
		send($socket, "bar\n", 0, $peer);
	}
}


$ cat test-udp-client.pl 
#!/usr/bin/perl

use warnings;
use strict;

use Socket;
use Time::HiRes qw/time/;

socket(my $socket, PF_INET, SOCK_DGRAM, 0)
	or die "socket: $!";
setsockopt($socket, SOL_SOCKET, SO_REUSEPORT, 1)
	or die "setsockopt: $!";
connect($socket, pack_sockaddr_in(9000, inet_aton("127.0.0.1")))
	or die "connect: $!";

my $peer;
my $msg = '';

my $s = '127.0.0.1:9000';
my $t = sprintf("%.6f", time());

print ">> ($s $t) foo\n";
send($socket, "foo\n", 0);

while ($peer = recv($socket, $msg, 65536, 0)) {

	chomp($msg);
	my ($port, $ip) = unpack_sockaddr_in($peer);
	my $s = inet_ntoa($ip) . ':' . $port;
        my $t = sprintf("%.6f", time());

	print "<< ($s $t) $msg\n";
}

Test run results on the server side:

$ perl test-udp-server.pl
<< (127.0.0.1:30152 1685228441.707326) foo
>> (127.0.0.1:30152 1685228441.707620) bar
>> (127.0.0.1:30152 1685228441.707737) bar
>> (127.0.0.1:30152 1685228441.707867) bar
>> (127.0.0.1:30152 1685228441.707949) bar
>> (127.0.0.1:30152 1685228441.708032) bar
>> (127.0.0.1:30152 1685228441.708108) bar
>> (127.0.0.1:30152 1685228441.708200) bar
>> (127.0.0.1:30152 1685228441.708290) bar
>> (127.0.0.1:30152 1685228441.708380) bar
>> (127.0.0.1:30152 1685228441.708453) bar

Note that 10 response packets are sent almost immediately. And here are the results on the client side:

$ perl test-udp-client.pl
>> (127.0.0.1:9000 1685228441.703600) foo
<< (127.0.0.1:9000 1685228441.710255) bar
<< (127.0.0.1:9000 1685228442.520890) bar
<< (127.0.0.1:9000 1685228443.382143) bar
<< (127.0.0.1:9000 1685228444.211470) bar
<< (127.0.0.1:9000 1685228445.073640) bar
<< (127.0.0.1:9000 1685228445.890434) bar
<< (127.0.0.1:9000 1685228446.720146) bar
<< (127.0.0.1:9000 1685228447.580390) bar
<< (127.0.0.1:9000 1685228448.409910) bar
<< (127.0.0.1:9000 1685228449.270287) bar

Note that the first response packet is sent immediately, and other packets are delayed as per proxy_download_rate.

If you still think there is a bug, please provide more details: notably, how do you test, and what do you observe, and what do you expect instead. A simple test script which demonstrates the problem would be awesome.

Note: See TracTickets for help on using tickets.