id summary reporter owner description type status priority milestone component version resolution keywords cc uname nginx_version 1602 use nginx stream module,use mutil worker_processes and UDP packet will be loss karbocheung@… "hello! i have some questin when use iperf3 test udp traffic. when ""worker_processes 1"" , all Datagrams successful. when ""worker_processes 6"" , many Datagrams loss. Please help me to check the problem,thx. ####INFO#### server and ip info: client(10.2.2.2:11000) -> nginx(9.10.10.10:11000) -> server(10.3.3.2:11000) nginx version: nginx/1.15.1 ####iperf3 testing process#### when i set nginx.conf -> worker_processes 1; client Sent 7735 datagram, also Recv 7735 datagram ,0/7735 0% loss. ####client command#### [root@client-01 ~]$ iperf3 -c 9.10.10.10 -p 11000 -u -t 10 -i 2 -b 1M -l 160 Connecting to host 9.10.10.10, port 11000 [ 4] local 10.2.2.2 port 39354 connected to 9.10.10.10 port 11000 [ ID] Interval Transfer Bandwidth Total Datagrams [ 4] 0.00-2.00 sec 232 KBytes 950 Kbits/sec 1485 [ 4] 2.00-4.00 sec 244 KBytes 1.00 Mbits/sec 1563 [ 4] 4.00-6.00 sec 244 KBytes 1000 Kbits/sec 1562 [ 4] 6.00-8.00 sec 244 KBytes 1.00 Mbits/sec 1563 [ 4] 8.00-10.00 sec 244 KBytes 1000 Kbits/sec 1562 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-10.00 sec 1.18 MBytes 990 Kbits/sec 0.008 ms 0/7735 (0%) [ 4] Sent 7735 datagrams iperf Done. ####server command#### [root@server-01 ~]$iperf3 -s -B 0.0.0.0 -p 11000 ----------------------------------------------------------- Server listening on 11000 ----------------------------------------------------------- Accepted connection from 9.10.10.10, port 33202 [ 5] local 10.3.3.2 port 11000 connected to 9.10.10.10 port 33468 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 110 KBytes 901 Kbits/sec 0.007 ms 0/704 (0%) [ 5] 1.00-2.00 sec 122 KBytes 1000 Kbits/sec 0.007 ms 0/781 (0%) [ 5] 2.00-3.00 sec 122 KBytes 1.00 Mbits/sec 0.006 ms 0/782 (0%) [ 5] 3.00-4.00 sec 122 KBytes 1000 Kbits/sec 0.008 ms 0/781 (0%) [ 5] 4.00-5.00 sec 122 KBytes 1000 Kbits/sec 0.007 ms 0/781 (0%) [ 5] 5.00-6.00 sec 122 KBytes 1000 Kbits/sec 0.007 ms 0/781 (0%) [ 5] 6.00-7.00 sec 122 KBytes 1.00 Mbits/sec 0.007 ms 0/782 (0%) [ 5] 7.00-8.00 sec 122 KBytes 1000 Kbits/sec 0.007 ms 0/781 (0%) [ 5] 8.00-9.00 sec 122 KBytes 1000 Kbits/sec 0.007 ms 0/781 (0%) [ 5] 9.00-10.00 sec 122 KBytes 1000 Kbits/sec 0.008 ms 0/781 (0%) [ 5] 10.00-10.04 sec 0.00 Bytes 0.00 bits/sec 0.008 ms 0/0 (-nan%) - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-10.04 sec 1.18 MBytes 986 Kbits/sec 0.008 ms 0/7735 (0%) ----------------------------------------------------------- Server listening on 11000 ----------------------------------------------------------- ####iperf3 testing process#### when i set nginx.conf -> worker_processes 6; client Sent 7735 datagram, also Recv 7735 datagram ,6415/7721 (83%) loss. ####client command#### [root@client-01 ~]$ iperf3 -c 9.10.10.10 -p 11000 -u -t 10 -i 2 -b 1M -l 160 Connecting to host 9.10.10.10, port 11000 [ 4] local 10.2.2.2 port 33597 connected to 9.10.10.10 port 11000 [ ID] Interval Transfer Bandwidth Total Datagrams [ 4] 0.00-2.00 sec 232 KBytes 950 Kbits/sec 1485 [ 4] 2.00-4.00 sec 244 KBytes 1.00 Mbits/sec 1563 [ 4] 4.00-6.00 sec 244 KBytes 1000 Kbits/sec 1562 [ 4] 6.00-8.00 sec 244 KBytes 1.00 Mbits/sec 1563 [ 4] 8.00-10.00 sec 244 KBytes 1000 Kbits/sec 1562 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-10.00 sec 1.18 MBytes 990 Kbits/sec 0.017 ms 6415/7721 (83%) [ 4] Sent 7721 datagrams iperf Done. ####server command#### [root@server-01 ~]$ iperf3 -s -B 0.0.0.0 -p 11000 ----------------------------------------------------------- Server listening on 11000 ----------------------------------------------------------- Accepted connection from 9.10.10.10, port 33325 [ 5] local 10.3.3.2 port 11000 connected to 9.10.10.10 port 53373 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 19.7 KBytes 161 Kbits/sec 0.033 ms 577/703 (82%) [ 5] 1.00-2.00 sec 30.5 KBytes 250 Kbits/sec 0.016 ms 571/766 (75%) [ 5] 2.00-3.00 sec 20.2 KBytes 165 Kbits/sec 0.053 ms 668/797 (84%) [ 5] 3.00-4.00 sec 17.8 KBytes 146 Kbits/sec 0.030 ms 655/769 (85%) [ 5] 4.00-5.00 sec 21.4 KBytes 175 Kbits/sec 0.021 ms 647/784 (83%) [ 5] 5.00-6.00 sec 16.2 KBytes 133 Kbits/sec 0.035 ms 686/790 (87%) [ 5] 6.00-7.00 sec 21.1 KBytes 173 Kbits/sec 0.032 ms 641/776 (83%) [ 5] 7.00-8.00 sec 14.8 KBytes 122 Kbits/sec 0.045 ms 662/757 (87%) [ 5] 8.00-9.00 sec 14.2 KBytes 116 Kbits/sec 0.060 ms 718/809 (89%) [ 5] 9.00-10.00 sec 28.1 KBytes 230 Kbits/sec 0.017 ms 590/770 (77%) [ 5] 10.00-10.04 sec 0.00 Bytes 0.00 bits/sec 0.017 ms 0/0 (-nan%) - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-10.04 sec 1.18 MBytes 986 Kbits/sec 0.017 ms 6415/7721 (83%) ----------------------------------------------------------- Server listening on 11000 ----------------------------------------------------------- #####nginx error.log##### [root@nginx-01 /data/home/root/nginx_1.5/logs]# tail -n10 error.log 2018/08/01 23:01:36 [error] 2782#0: *15147 sendmsg() failed (111: Connection refused) while proxying and sending to upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: ""10.3.3.2:11000"", bytes from/to client:2080/0, bytes from/to upstream:0/1920 2018/08/01 23:01:37 [error] 2783#0: *15153 recv() failed (111: Connection refused) while proxying and reading from upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: ""10.3.3.2:11000"", bytes from/to client:23520/0, bytes from/to upstream:0/23520 2018/08/01 23:01:38 [error] 2780#0: *15155 recv() failed (111: Connection refused) while proxying and reading from upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: ""10.3.3.2:11000"", bytes from/to client:33440/0, bytes from/to upstream:0/33440 2018/08/01 23:01:39 [error] 2781#0: *15148 sendmsg() failed (111: Connection refused) while proxying and sending to upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: ""10.3.3.2:11000"", bytes from/to client:61920/0, bytes from/to upstream:0/61760 2018/08/01 23:01:40 [error] 2780#0: *15155 sendmsg() failed (111: Connection refused) while proxying and sending to upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: ""10.3.3.2:11000"", bytes from/to client:81280/0, bytes from/to upstream:0/81120 2018/08/01 23:01:41 [error] 2782#0: *15156 recv() failed (111: Connection refused) while proxying and reading from upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: ""10.3.3.2:11000"", bytes from/to client:94560/0, bytes from/to upstream:0/94560 2018/08/01 23:01:42 [error] 2783#0: *15153 sendmsg() failed (111: Connection refused) while proxying and sending to upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: ""10.3.3.2:11000"", bytes from/to client:131360/0, bytes from/to upstream:0/131200 2018/08/01 23:01:43 [error] 2783#0: *15163 sendmsg() failed (111: Connection refused) while proxying and sending to upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: ""10.3.3.2:11000"", bytes from/to client:25280/0, bytes from/to upstream:0/25120 2018/08/01 23:01:44 [error] 2780#0: *15161 recv() failed (111: Connection refused) while proxying and reading from upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: ""10.3.3.2:11000"", bytes from/to client:81280/0, bytes from/to upstream:0/81280 2018/08/01 23:01:45 [error] 2783#0: *15165 sendmsg() failed (111: Connection refused) while proxying and sending to upstream, udp client: 10.2.2.2, server: 0.0.0.0:11000, upstream: ""10.3.3.2:11000"", bytes from/to client:47360/0, bytes from/to upstream:0/47200 #####tcpdump info##### use tcpdump capture the package, i find nginx use 19 ports, and every session count not the same. [root@nginx-01 ~]#tcpdump -i any port 11001 -nnn > 2.txt [root@nginx-01 ~]# awk '$3 ~ /9.10.10.10/{print $3}' /data/home/root/2.txt |sort|uniq -c 32719 9.10.10.10.34846 820 9.10.10.10.36089 42751 9.10.10.10.36347 100 9.10.10.10.36521 82 9.10.10.10.37650 24 9.10.10.10.39291 129 9.10.10.10.39542 19 9.10.10.10.41983 500 9.10.10.10.42853 9878 9.10.10.10.43915 21 9.10.10.10.44628 671 9.10.10.10.44874 8128 9.10.10.10.45933 348 9.10.10.10.51730 95 9.10.10.10.55548 361 9.10.10.10.56743 512 9.10.10.10.58357 184 9.10.10.10.58739 550 9.10.10.10.58743 ####https://nginx.org/en/CHANGES#### Changes with nginx 1.15.0 05 Jun 2018 *) Change: the ""ssl"" directive is deprecated; the ""ssl"" parameter of the ""listen"" directive should be used instead. *) Change: now nginx detects missing SSL certificates during configuration testing when using the ""ssl"" parameter of the ""listen"" directive. *) Feature: now the stream module can handle multiple incoming UDP datagrams from a client within a single session. *) Bugfix: it was possible to specify an incorrect response code in the ""proxy_cache_valid"" directive. the stream module can handle multiple incoming UDP datagrams from a client within a single session. but i use 6 core, a single session use mutil core,so use iperf3 test udp will be loss." defect closed major nginx-core 1.15.x invalid stream module nginx/1.15.1