load_balance failed in ngx_stream_proxy_module because of "pending buffers"

[qleeAI@…]

If s->connection->read->ready is true when entering ngx_stream_proxy_module, ngx_stream_proxy_downstream_handler is called and read some data , stored in u->upstream_out.

If connect to upstream failed and entering ngx_stream_proxy_next_upstream() to connect to next upstrem server. u->upstream_out is not NULL and stream session is finalized.

In function ngx_stream_proxy_next_upstream

    if (u->upstream_out || u->upstream_busy || (pc && pc->buffered)) {
        ngx_log_error(NGX_LOG_ERR, s->connection->log, 0,
                      "pending buffers on next upstream");
        ngx_stream_proxy_finalize(s, NGX_STREAM_INTERNAL_SERVER_ERROR);
        return;
    }

Example

Obviously the example only used to descript the problem, we don't use it like this on production environment.

Nginx config file like this.

daemon off;
master_process off;

events {
    worker_connections  1024;
}

stream {
    upstream backend {
        server 10.10.10.10:80;
        server 115.231.25.133:80;
    }
    server {
        listen 80;
        ssl_preread on;
        preread_buffer_size 64;
        proxy_pass backend;
        proxy_connect_timeout 3s;
    }
}

Using curl to send a HTTP request, Nginx only try the first server.

error.log

2017/07/12 22:43:37 [error] 25361#0: *1 upstream timed out (110: Connection timed out) while connecting to upstream, client: 127.0.0.1, server: 0.0.0.0:80, upstream: "10.10.10.10:80", bytes from/to client:9/0, bytes from/to upstream:0/0
2017/07/12 22:43:37 [error] 25361#0: *1 pending buffers on next upstream while connecting to upstream, client: 127.0.0.1, server: 0.0.0.0:80, upstream: "10.10.10.10:80", bytes from/to client:9/0, bytes from/to upstream:0/0

[Roman Arutyunyan]

Thanks for reporting this. Please try the attached patch.

[qleeAI@…]

Replying to arut:

Thanks for reporting this. Please try the attached patch.

This patch works for me. Thanks.

[Andrey Zelenkov <zelenkov@…>]

In 1199:08f6eacf1cfe/nginx-tests:

Tests: stream proxy next upstream with ssl_preread (ticket #1317).

Ensure that next TCP upstream can be selected with pending buffers.

[Roman Arutyunyan <arut@…>]

In 7098:7bfbf73db920/nginx:

Stream: relaxed next upstream condition (ticket #1317).

When switching to a next upstream, some buffers could be stuck in the middle
of the filter chain. A condition existed that raised an error when this
happened. As it turned out, this condition prevented switching to a next
upstream if ssl preread was used with the TCP protocol (see the ticket).

In fact, the condition does not make sense for TCP, since after successful
connection to an upstream switching to another upstream never happens. As for
UDP, the issue with stuck buffers is unlikely to happen, but is still possible.
Specifically, if a filter delays sending data to upstream.

The condition can be relaxed to only check the "buffered" bitmask of the
upstream connection. The new condition is simpler and fixes the ticket issue
as well. Additionally, the upstream_out chain is now reset for UDP prior to
connecting to a new upstream to prevent repeating the client data twice.

[Roman Arutyunyan <arut@…>]

In 7142:b9d919b53593/nginx:

Stream: relaxed next upstream condition (ticket #1317).

When switching to a next upstream, some buffers could be stuck in the middle
of the filter chain. A condition existed that raised an error when this
happened. As it turned out, this condition prevented switching to a next
upstream if ssl preread was used with the TCP protocol (see the ticket).

In fact, the condition does not make sense for TCP, since after successful
connection to an upstream switching to another upstream never happens. As for
UDP, the issue with stuck buffers is unlikely to happen, but is still possible.
Specifically, if a filter delays sending data to upstream.

The condition can be relaxed to only check the "buffered" bitmask of the
upstream connection. The new condition is simpler and fixes the ticket issue
as well. Additionally, the upstream_out chain is now reset for UDP prior to
connecting to a new upstream to prevent repeating the client data twice.