use nginx stream module,use mutil worker_processes and UDP packet will be loss

[karbocheung@…]

hello!
i have some questin when use iperf3 test udp traffic.
when "worker_processes 1" , all Datagrams successful.
when "worker_processes 6" , many Datagrams loss.
Please help me to check the problem,thx.

####INFO####
server and ip info:
client(10.2.2.2:11000) -> nginx(9.10.10.10:11000) -> server(10.3.3.2:11000)

nginx version:
nginx/1.15.1

####iperf3 testing process####
when i set nginx.conf -> worker_processes 1;
client Sent 7735 datagram, also Recv 7735 datagram ,0/7735 0% loss.

####client command####
[root@client-01 ~]$ iperf3 -c 9.10.10.10 -p 11000 -u -t 10 -i 2 -b 1M -l 160
Connecting to host 9.10.10.10, port 11000
[ 4] local 10.2.2.2 port 39354 connected to 9.10.10.10 port 11000
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-2.00 sec 232 KBytes 950 Kbits/sec 1485
[ 4] 2.00-4.00 sec 244 KBytes 1.00 Mbits/sec 1563
[ 4] 4.00-6.00 sec 244 KBytes 1000 Kbits/sec 1562
[ 4] 6.00-8.00 sec 244 KBytes 1.00 Mbits/sec 1563
[ 4] 8.00-10.00 sec 244 KBytes 1000 Kbits/sec 1562

• - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-10.00 sec 1.18 MBytes 990 Kbits/sec 0.008 ms 0/7735 (0%)
[ 4] Sent 7735 datagrams

iperf Done.

####server command####
[root@server-01 ~]$iperf3 -s -B 0.0.0.0 -p 11000

---

Server listening on 11000

---

Accepted connection from 9.10.10.10, port 33202
[ 5] local 10.3.3.2 port 11000 connected to 9.10.10.10 port 33468
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 110 KBytes 901 Kbits/sec 0.007 ms 0/704 (0%)
[ 5] 1.00-2.00 sec 122 KBytes 1000 Kbits/sec 0.007 ms 0/781 (0%)
[ 5] 2.00-3.00 sec 122 KBytes 1.00 Mbits/sec 0.006 ms 0/782 (0%)
[ 5] 3.00-4.00 sec 122 KBytes 1000 Kbits/sec 0.008 ms 0/781 (0%)
[ 5] 4.00-5.00 sec 122 KBytes 1000 Kbits/sec 0.007 ms 0/781 (0%)
[ 5] 5.00-6.00 sec 122 KBytes 1000 Kbits/sec 0.007 ms 0/781 (0%)
[ 5] 6.00-7.00 sec 122 KBytes 1.00 Mbits/sec 0.007 ms 0/782 (0%)
[ 5] 7.00-8.00 sec 122 KBytes 1000 Kbits/sec 0.007 ms 0/781 (0%)
[ 5] 8.00-9.00 sec 122 KBytes 1000 Kbits/sec 0.007 ms 0/781 (0%)
[ 5] 9.00-10.00 sec 122 KBytes 1000 Kbits/sec 0.008 ms 0/781 (0%)
[ 5] 10.00-10.04 sec 0.00 Bytes 0.00 bits/sec 0.008 ms 0/0 (-nan%)

• - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-10.04 sec 1.18 MBytes 986 Kbits/sec 0.008 ms 0/7735 (0%)

---

Server listening on 11000

---

####iperf3 testing process####
when i set nginx.conf -> worker_processes 6;
client Sent 7735 datagram, also Recv 7735 datagram ,6415/7721 (83%) loss.

####client command####
[root@client-01 ~]$ iperf3 -c 9.10.10.10 -p 11000 -u -t 10 -i 2 -b 1M -l 160
Connecting to host 9.10.10.10, port 11000
[ 4] local 10.2.2.2 port 33597 connected to 9.10.10.10 port 11000
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-2.00 sec 232 KBytes 950 Kbits/sec 1485
[ 4] 2.00-4.00 sec 244 KBytes 1.00 Mbits/sec 1563
[ 4] 4.00-6.00 sec 244 KBytes 1000 Kbits/sec 1562
[ 4] 6.00-8.00 sec 244 KBytes 1.00 Mbits/sec 1563
[ 4] 8.00-10.00 sec 244 KBytes 1000 Kbits/sec 1562

• - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-10.00 sec 1.18 MBytes 990 Kbits/sec 0.017 ms 6415/7721 (83%)
[ 4] Sent 7721 datagrams

iperf Done.

####server command####
[root@server-01 ~]$ iperf3 -s -B 0.0.0.0 -p 11000

---

Server listening on 11000

---

Accepted connection from 9.10.10.10, port 33325
[ 5] local 10.3.3.2 port 11000 connected to 9.10.10.10 port 53373
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 19.7 KBytes 161 Kbits/sec 0.033 ms 577/703 (82%)
[ 5] 1.00-2.00 sec 30.5 KBytes 250 Kbits/sec 0.016 ms 571/766 (75%)
[ 5] 2.00-3.00 sec 20.2 KBytes 165 Kbits/sec 0.053 ms 668/797 (84%)
[ 5] 3.00-4.00 sec 17.8 KBytes 146 Kbits/sec 0.030 ms 655/769 (85%)
[ 5] 4.00-5.00 sec 21.4 KBytes 175 Kbits/sec 0.021 ms 647/784 (83%)
[ 5] 5.00-6.00 sec 16.2 KBytes 133 Kbits/sec 0.035 ms 686/790 (87%)
[ 5] 6.00-7.00 sec 21.1 KBytes 173 Kbits/sec 0.032 ms 641/776 (83%)
[ 5] 7.00-8.00 sec 14.8 KBytes 122 Kbits/sec 0.045 ms 662/757 (87%)
[ 5] 8.00-9.00 sec 14.2 KBytes 116 Kbits/sec 0.060 ms 718/809 (89%)
[ 5] 9.00-10.00 sec 28.1 KBytes 230 Kbits/sec 0.017 ms 590/770 (77%)
[ 5] 10.00-10.04 sec 0.00 Bytes 0.00 bits/sec 0.017 ms 0/0 (-nan%)

• - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-10.04 sec 1.18 MBytes 986 Kbits/sec 0.017 ms 6415/7721 (83%)

---

Server listening on 11000

---

#####nginx error.log#####
[root@nginx-01 /data/home/root/nginx_1.5/logs]# tail -n10 error.log
2018/08/01 23:01:36 [error] 2782#0: *15147 sendmsg() failed (111: Connection refused) while proxying and sending to upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: "10.3.3.2:11000", bytes from/to client:2080/0, bytes from/to upstream:0/1920
2018/08/01 23:01:37 [error] 2783#0: *15153 recv() failed (111: Connection refused) while proxying and reading from upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: "10.3.3.2:11000", bytes from/to client:23520/0, bytes from/to upstream:0/23520
2018/08/01 23:01:38 [error] 2780#0: *15155 recv() failed (111: Connection refused) while proxying and reading from upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: "10.3.3.2:11000", bytes from/to client:33440/0, bytes from/to upstream:0/33440
2018/08/01 23:01:39 [error] 2781#0: *15148 sendmsg() failed (111: Connection refused) while proxying and sending to upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: "10.3.3.2:11000", bytes from/to client:61920/0, bytes from/to upstream:0/61760
2018/08/01 23:01:40 [error] 2780#0: *15155 sendmsg() failed (111: Connection refused) while proxying and sending to upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: "10.3.3.2:11000", bytes from/to client:81280/0, bytes from/to upstream:0/81120
2018/08/01 23:01:41 [error] 2782#0: *15156 recv() failed (111: Connection refused) while proxying and reading from upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: "10.3.3.2:11000", bytes from/to client:94560/0, bytes from/to upstream:0/94560
2018/08/01 23:01:42 [error] 2783#0: *15153 sendmsg() failed (111: Connection refused) while proxying and sending to upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: "10.3.3.2:11000", bytes from/to client:131360/0, bytes from/to upstream:0/131200
2018/08/01 23:01:43 [error] 2783#0: *15163 sendmsg() failed (111: Connection refused) while proxying and sending to upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: "10.3.3.2:11000", bytes from/to client:25280/0, bytes from/to upstream:0/25120
2018/08/01 23:01:44 [error] 2780#0: *15161 recv() failed (111: Connection refused) while proxying and reading from upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: "10.3.3.2:11000", bytes from/to client:81280/0, bytes from/to upstream:0/81280
2018/08/01 23:01:45 [error] 2783#0: *15165 sendmsg() failed (111: Connection refused) while proxying and sending to upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: "10.3.3.2:11000", bytes from/to client:47360/0, bytes from/to upstream:0/47200

#####tcpdump info#####

use tcpdump capture the package, i find nginx use 19 ports, and every session count not the same.

[root@nginx-01 ~]#tcpdump -i any port 11001 -nnn > 2.txt
[root@nginx-01 ~]# awk '$3 ~ /9.10.10.10/{print $3}' /data/home/root/2.txt |sort|uniq -c
32719 9.10.10.10.34846
820 9.10.10.10.36089
42751 9.10.10.10.36347
100 9.10.10.10.36521
82 9.10.10.10.37650
24 9.10.10.10.39291
129 9.10.10.10.39542
19 9.10.10.10.41983
500 9.10.10.10.42853
9878 9.10.10.10.43915
21 9.10.10.10.44628
671 9.10.10.10.44874
8128 9.10.10.10.45933
348 9.10.10.10.51730
95 9.10.10.10.55548
361 9.10.10.10.56743
512 9.10.10.10.58357
184 9.10.10.10.58739
550 9.10.10.10.58743

####​https://nginx.org/en/CHANGES####
Changes with nginx 1.15.0 05 Jun 2018

*) Change: the "ssl" directive is deprecated; the "ssl" parameter of the

"listen" directive should be used instead.

*) Change: now nginx detects missing SSL certificates during

configuration testing when using the "ssl" parameter of the "listen"
directive.

*) Feature: now the stream module can handle multiple incoming UDP

datagrams from a client within a single session.

*) Bugfix: it was possible to specify an incorrect response code in the

"proxy_cache_valid" directive.

the stream module can handle multiple incoming UDP datagrams from a client within a single session.
but i use 6 core, a single session use mutil core,so use iperf3 test udp will be loss.

[Maxim Dounin]

Matching of UDP packets to an existing session is only guaranteed within a single worker process. If your protocol requires a single session (as the provided tests seems to do), you should either use only one worker processes (worker_processes 1;) or use listen ... reuseport. See d27aa9060c95 for more details.