Opened 3 years ago

Closed 2 years ago

#1298 closed enhancement (fixed)

when use nginx stream module to proxy UDP traffic, one connection connect to nginx, but nginx will create one new socket to send every UDP packet

Reported by: cheng0201@… Owned by:
Priority: major Milestone:
Component: nginx-core Version: 1.12.x
Keywords: Cc:
uname -a: Linux centos 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.12.0
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC)
built with OpenSSL 1.1.0e 16 Feb 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-openssl=openssl-1.1.0e --add-module=modules/nginx_http_status_module --add-module=modules/nginx_http_upstream_check_module --add-module=modules/nginx-sticky-module-ng --add-module=modules/nginx_stream_upstream_check_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC -DNGX_HTTP_STATUS -DNGX_HTTP_UPSTREAM_CHECK -DNGX_STREAM_UPSTREAM_CHECK' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

Description

just use netperf to test nginx stream module UDP proxy performance, a host as upstream server to run netserver, and another host run netperf to send traffic nginx as below:
netperf -t UDP_STREAM -c -C -fM -n4 -H 172.16.0.15 -l 60 -- -m 1400 -P ,10000
nginx stream part config as below:
stream {

access_log /var/log/lb/stream-access.log stream_format;

tcp_nodelay on;
proxy_connect_timeout 5s;
proxy_timeout 60s;

upstream backend-mxx3yddtxa {

server 172.16.0.19:12865 weight=10 max_fails=0;

}

upstream backend-39xka0qmti {

server 172.16.0.19:10000 weight=10 max_fails=0;

}

server {

listen 172.16.0.15:12865 reuseport;

proxy_timeout 86400s;
proxy_pass backend-mxx3yddtxa;

}

server {

listen 172.16.0.15:10000 udp reuseport;

#proxy_timeout 0s;
proxy_responses 0;
proxy_pass backend-39xka0qmti;

}

}

whatever set proxy_timeout 0s or proxy_responses 0, nginx will use up all the local udp port, and at last continue to print out error log as below:
2017/06/22 06:50:26 [error] 21260#21260: *138915386 connect() to 172.16.0.19:10000 failed (11: Resource temporarily unavailable) while connecting to upstream, udp c
lient: 172.16.0.18, server: 172.16.0.15:10000, upstream: "172.16.0.19:10000", bytes from/to client:1400/0, bytes from/to upstream:0/0
as we know, UDP is connectionless protocol, and nginx receive incoming UDP packet through listen socket and without new udp socket, so forward udp packet to upstream without connection information. when we create a new socket for every UDP packet, socket will easily used up.
Maybe we should add a config option, when enable this option, attach a map with upstream, and the map use remote IP and port as key, and the socket used to forward to upstream as value, when first remote UDP come to nginx, nginx insert a entry for it, later UDP packet will find this map, if no entry match, new a entry and insert it, else will get stored socket send the packet.
of coure, maybe there are some other good way to solve this issue.

Attachments (1)

udp_session_reuse.diff (14.0 KB ) - added by cheng0201@… 3 years ago.
udp session reuse diff

Download all attachments as: .zip

Change History (6)

comment:1 by Roman Arutyunyan, 3 years ago

Currently, nginx creates a new socket for proxying every incoming UDP datagram.

Having a single socket for each client sounds like a good idea, but the problem is UDP datagrams can arrive to different nginx workers. There are several workarounds to fix that, but none of them is perfect enough.

by cheng0201@…, 3 years ago

Attachment: udp_session_reuse.diff added

udp session reuse diff

comment:2 by cheng0201@…, 3 years ago

while use nginx with a single worker, there is a workaround to reuse UDP session to forward UDP packets to upstream. it could reduce socket used with multiply workers, but of course it's not a perfect solution. just for your interest, and hope it's a little helpful.

comment:3 by cheng0201@…, 3 years ago

this diff is for nginx 1.12.0

in reply to:  2 comment:4 by sepherosa@…, 3 years ago

Replying to cheng0201@…:

while use nginx with a single worker, there is a workaround to reuse UDP session to forward UDP packets to upstream. it could reduce socket used with multiply workers, but of course it's not a perfect solution. just for your interest, and hope it's a little helpful.

You can enable reuseport for the UDP listen socket. I think even on Linux, the UDP 4-tuple will be hashed properly to a specific worker; and along w/ your patch, it should work.

comment:5 by vl, 2 years ago

Resolution: fixed
Status: newclosed

UDP handling was improved in in nginx-1.15.0 (http://hg.nginx.org/nginx/rev/d27aa9060c95)

Note: See TracTickets for help on using tickets.