Opened 12 months ago

Closed 10 months ago

Last modified 8 months ago

#1317 closed defect (fixed)

load_balance failed in ngx_stream_proxy_module because of "pending buffers"

Reported by: qleeAI@… Owned by:
Priority: minor Milestone: 1.13
Component: nginx-module Version: 1.13.x
Keywords: stream proxy load_balance Cc: gcc, (Ubuntu, 5.4.0-6ubuntu1~16.04.4), 5.4.0, 20160609
uname -a: Linux kenan-desktop 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.13.4 built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) configure arguments: --with-debug --with-stream --with-stream_ssl_preread_module --with-pcre --with-zlib=../zlib-1.2.8

Description

If s->connection->read->ready is true when entering ngx_stream_proxy_module, ngx_stream_proxy_downstream_handler is called and read some data , stored in u->upstream_out.

If connect to upstream failed and entering ngx_stream_proxy_next_upstream() to connect to next upstrem server. u->upstream_out is not NULL and stream session is finalized.

In function ngx_stream_proxy_next_upstream

    if (u->upstream_out || u->upstream_busy || (pc && pc->buffered)) {
        ngx_log_error(NGX_LOG_ERR, s->connection->log, 0,
                      "pending buffers on next upstream");
        ngx_stream_proxy_finalize(s, NGX_STREAM_INTERNAL_SERVER_ERROR);
        return;
    }

Example

Obviously the example only used to descript the problem, we don't use it like this on production environment.

Nginx config file like this.

daemon off;
master_process off;

events {
    worker_connections  1024;
}

stream {
    upstream backend {
        server 10.10.10.10:80;
        server 115.231.25.133:80;
    }
    server {
        listen 80;
        ssl_preread on;
        preread_buffer_size 64;
        proxy_pass backend;
        proxy_connect_timeout 3s;
    }
}

Using curl to send a HTTP request, Nginx only try the first server.

error.log

2017/07/12 22:43:37 [error] 25361#0: *1 upstream timed out (110: Connection timed out) while connecting to upstream, client: 127.0.0.1, server: 0.0.0.0:80, upstream: "10.10.10.10:80", bytes from/to client:9/0, bytes from/to upstream:0/0
2017/07/12 22:43:37 [error] 25361#0: *1 pending buffers on next upstream while connecting to upstream, client: 127.0.0.1, server: 0.0.0.0:80, upstream: "10.10.10.10:80", bytes from/to client:9/0, bytes from/to upstream:0/0

Attachments (1)

stream-udp-next-upstream (1.2 KB) - added by arut 11 months ago.

Download all attachments as: .zip

Change History (7)

Changed 11 months ago by arut

comment:1 follow-up: Changed 11 months ago by arut

Thanks for reporting this. Please try the attached patch.

comment:2 in reply to: ↑ 1 Changed 11 months ago by qleeAI@…

Replying to arut:

Thanks for reporting this. Please try the attached patch.

This patch works for me. Thanks.

comment:3 Changed 11 months ago by Andrey Zelenkov <zelenkov@…>

In 1199:08f6eacf1cfe/nginx-tests:

Tests: stream proxy next upstream with ssl_preread (ticket #1317).

Ensure that next TCP upstream can be selected with pending buffers.

comment:4 Changed 10 months ago by Roman Arutyunyan <arut@…>

In 7098:7bfbf73db920/nginx:

Stream: relaxed next upstream condition (ticket #1317).

When switching to a next upstream, some buffers could be stuck in the middle
of the filter chain. A condition existed that raised an error when this
happened. As it turned out, this condition prevented switching to a next
upstream if ssl preread was used with the TCP protocol (see the ticket).

In fact, the condition does not make sense for TCP, since after successful
connection to an upstream switching to another upstream never happens. As for
UDP, the issue with stuck buffers is unlikely to happen, but is still possible.
Specifically, if a filter delays sending data to upstream.

The condition can be relaxed to only check the "buffered" bitmask of the
upstream connection. The new condition is simpler and fixes the ticket issue
as well. Additionally, the upstream_out chain is now reset for UDP prior to
connecting to a new upstream to prevent repeating the client data twice.

comment:5 Changed 10 months ago by arut

  • Resolution set to fixed
  • Status changed from new to closed

comment:6 Changed 8 months ago by Roman Arutyunyan <arut@…>

In 7142:b9d919b53593/nginx:

Stream: relaxed next upstream condition (ticket #1317).

When switching to a next upstream, some buffers could be stuck in the middle
of the filter chain. A condition existed that raised an error when this
happened. As it turned out, this condition prevented switching to a next
upstream if ssl preread was used with the TCP protocol (see the ticket).

In fact, the condition does not make sense for TCP, since after successful
connection to an upstream switching to another upstream never happens. As for
UDP, the issue with stuck buffers is unlikely to happen, but is still possible.
Specifically, if a filter delays sending data to upstream.

The condition can be relaxed to only check the "buffered" bitmask of the
upstream connection. The new condition is simpler and fixes the ticket issue
as well. Additionally, the upstream_out chain is now reset for UDP prior to
connecting to a new upstream to prevent repeating the client data twice.

Note: See TracTickets for help on using tickets.