With UDP load balancing proxy protocol is not appended to every separate UDP packet

[silh@…]

In our setup nginx is used to load-balance MQTT-SN traffic which is sent over UDP. The server behind the proxy expects packets to contain PROXY protocol information in order to use it for further communication with the sender.
Our config looks as follows:

load_module modules/ngx_stream_js_module.so;

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

stream {
    include conf.d/mqttsn_back_route.udp;
    include conf.d/mqttsn_forward_route.udp;
}

where mqttsn_back_route.udp is:

  js_include conf.d/udp-sngw-backroute.js;
  js_set     $destination setDestination;
  
  server {
    listen 1885 udp;
    error_log  /var/log/nginx/udp_error.log debug;
    preread_buffer_size 1k;
    js_preread getDestination;
    js_filter filterResponse;
    proxy_responses 0;
    proxy_pass $destination;
    proxy_connect_timeout 1s;
    proxy_bind 0.0.0.0:1883;
  }

and conf.d/mqttsn_forward_route.udp is:

  upstream udp_sngateway_4444  {
    server router00:4444 max_fails=1002 fail_timeout=30s;
    server router01:4444 max_fails=2 fail_timeout=10s;
  }
  server {
    listen 1883 udp;
    error_log  /var/log/nginx/udp_error.log debug;
    proxy_bind 10.3.201.20;
    #proxy_bind $server_addr;
    proxy_protocol on;
    proxy_responses 1;
    proxy_pass udp_sngateway_4444;
    proxy_buffer_size 640k;
    proxy_timeout 8s;
    preread_buffer_size 10k;
    js_filter dropIfPong;
  }

Mentioned js script:

	
var destination
var cleanResponse
var line = '';
function getDestination(s) {
    s.on('upload', function (data, flags) {
        var n = data.length;
        if (n != -1) {
            line = data.substr(0, n);
            splitDestination(s);
            s.done();
        }
    });
}
function splitDestination(s) {
    var parts = line.split("\r\n");
    var header = parts[0].split(" ");
    var sig = header[0];
    var upstreamAddress = header[3];
    var upstreamPort = header[5];
    if (parts[1] === undefined) {
        throw "proxy protocol: no payload";
    }
    cleanResponse = parts[1];
    if(sig != "PROXY") {
        throw "proxy protocol: invalid signature received";
    }
    destination = upstreamAddress + ":" + upstreamPort;
}
function dropIfPong(s) {
        s.on('download', function(data, flags) {
                if (data.startsWith("MESSAGE-ROUTER-ACK")) {
                        return s.ERROR;
                }
                return s.OK;
        });

function setDestination(s) {
    return destination;
}
function filterResponse(s) {
    s.on('upload', function(data, flags) {
        s.send(cleanResponse,flags);
        s.off('upload');
    });
}

The datagram packets that reachs the server sometimes doesn't contain the Proxy protocol in it which leads to a problem on the server side. According to the comment here - https://trac.nginx.org/nginx/ticket/1543 - every packet is expected to have that header nginx 1.11.5.
We have captured the traffic with tcpdump and I can provide tan example of correct and incorrect packets (they are attached to the issue).

[vl]

Unfortunately, there is nothing to add to mentioned comment in ticket 1543:

"Note well that the PROXY protocol specification doesn't really define how UDP should be handled, so it might not be a good idea to use it with UDP."

attempts to implement it would require making a lot of ad-hoc decisions and will
bring unnecessary additional complexity.