when use nginx stream module to proxy UDP traffic, one connection connect to nginx, but nginx will create one new socket to send every UDP packet

[cheng0201@…]

just use netperf to test nginx stream module UDP proxy performance, a host as upstream server to run netserver, and another host run netperf to send traffic nginx as below:
netperf -t UDP_STREAM -c -C -fM -n4 -H 172.16.0.15 -l 60 -- -m 1400 -P ,10000
nginx stream part config as below：
stream {

access_log /var/log/lb/stream-access.log stream_format;

tcp_nodelay on;
proxy_connect_timeout 5s;
proxy_timeout 60s;

upstream backend-mxx3yddtxa {

server 172.16.0.19:12865 weight=10 max_fails=0;

}

upstream backend-39xka0qmti {

server 172.16.0.19:10000 weight=10 max_fails=0;

}

server {

listen 172.16.0.15:12865 reuseport;

proxy_timeout 86400s;
proxy_pass backend-mxx3yddtxa;

}

server {

listen 172.16.0.15:10000 udp reuseport;

#proxy_timeout 0s;
proxy_responses 0;
proxy_pass backend-39xka0qmti;

}

}

whatever set proxy_timeout 0s or proxy_responses 0, nginx will use up all the local udp port, and at last continue to print out error log as below:
2017/06/22 06:50:26 [error] 21260#21260: *138915386 connect() to 172.16.0.19:10000 failed (11: Resource temporarily unavailable) while connecting to upstream, udp c
lient: 172.16.0.18, server: 172.16.0.15:10000, upstream: "172.16.0.19:10000", bytes from/to client:1400/0, bytes from/to upstream:0/0
as we know, UDP is connectionless protocol, and nginx receive incoming UDP packet through listen socket and without new udp socket, so forward udp packet to upstream without connection information. when we create a new socket for every UDP packet, socket will easily used up.
Maybe we should add a config option, when enable this option, attach a map with upstream, and the map use remote IP and port as key, and the socket used to forward to upstream as value, when first remote UDP come to nginx, nginx insert a entry for it, later UDP packet will find this map, if no entry match, new a entry and insert it, else will get stored socket send the packet.
of coure, maybe there are some other good way to solve this issue.

[Roman Arutyunyan]

Currently, nginx creates a new socket for proxying every incoming UDP datagram.

Having a single socket for each client sounds like a good idea, but the problem is UDP datagrams can arrive to different nginx workers. There are several workarounds to fix that, but none of them is perfect enough.

[cheng0201@…]

udp session reuse diff

[cheng0201@…]

while use nginx with a single worker, there is a workaround to reuse UDP session to forward UDP packets to upstream. it could reduce socket used with multiply workers, but of course it's not a perfect solution. just for your interest, and hope it's a little helpful.

[cheng0201@…]

this diff is for nginx 1.12.0

[sepherosa@…]

Replying to cheng0201@…:

while use nginx with a single worker, there is a workaround to reuse UDP session to forward UDP packets to upstream. it could reduce socket used with multiply workers, but of course it's not a perfect solution. just for your interest, and hope it's a little helpful.

You can enable reuseport for the UDP listen socket. I think even on Linux, the UDP 4-tuple will be hashed properly to a specific worker; and along w/ your patch, it should work.

[vl]

UDP handling was improved in in nginx-1.15.0 (​http://hg.nginx.org/nginx/rev/d27aa9060c95)