ngx_http_upstream_process_non_buffered_request do_write Dead-loop

[bullerdu@…]

In non-buffering transport mode(proxy_request_buffering off), I encountered a write dead loop problem with the following code path:

src/http/ngx_http_upstream.c:3666 ngx_http_upstream_process_non_buffered_request
  ->
     src/http/ngx_http_upstream.c:3691
     rc = ngx_http_output_filter(r, u->out_bufs);  /* u->out_bufs=0x0 */
         ->
         src/http/ngx_http_write_filter_module.c:48
         ngx_http_write_filter
            ->
            src/http/ngx_http_write_filter_module.c:295
            chain = c->send_chain(c, r->out, limit);
               src/os/uinx/ngx_linux_sendfile_chain.c:50
               ngx_linux_sendfile_chain
               /* code here return
                  if (!wev->ready) {
                      return in;
                  } 
               */

             src/http/ngx_http_write_filter_module.c:342
             /* code here return
                if (chain) {
                    c->buffered |= NGX_HTTP_WRITE_BUFFERED;
                    return NGX_AGAIN;
                }
              */

src/http/ngx_http_upstream.c:3735
n = upstream->recv(upstream, b->last, size);
/* code here, return n = -1 */

->

src/http/ngx_http_upstream.c:3753
/* code here, continue */

-> 

src/http/ngx_http_upstream.c:3691
rc = ngx_http_output_filter(r, u->out_bufs);

My analysis is as follows:

First:
when the downstream is written, ngx_linux_sendfile_chain calls ngx_writev and returns NGX_AGAIN, setting wev->ready =0
src/os/uinx/ngx_linux_sendfile_chain.c:204
if (n == NGX_AGAIN) {
wev->ready = 0;
return in;
}

Second:
src/http/ngx_http_upstream.c:3691
ngx_http_output_filter returns NGX_AGAIN, u->out_bufs = NULL, u->busy_bufs! = NULL

size > 0 && upstream->read->ready = 1, upstream->recv returns -1, So I keep writing downstream(ngx_http_output_filter), but downstream->write->ready = 0 so ngx_http_output_filter keeps returning -2

The resulting problems：
ngx_http_output_filter loop call, some filter_modules have cpu consumption problem, CPU 100%

The Solution：

Check the return value of ngx_http_output_filter. If it is NGX_AGAIN, break the write loop

src/http/ngx_http_upstream.c:3691

rc = ngx_http_output_filter(r, u->out_bufs);
if (rc == NGX_ERROR) {

ngx_http_upstream_finalize_request(r, u, NGX_ERROR);
return;

}
if (rc == NGX_AGAIN) {

...

break;

}

[Maxim Dounin]

The loop is expected to stop once the u->buffer is full. Could you please clarify the configuration you are seeing the problem with?

[bullerdu@…]

Replying to Maxim Dounin:

The loop is expected to stop once the u->buffer is full. Could you please clarify the configuration you are seeing the problem with?

First of all, u->buffer is not full in my scenario. And n = upstream->recv(upstream, b->last, size); Always return -1.

Because the nginx production environment configuration is very complicated, it cannot be provided. And this phenomenon is not easy to repeat, only occasionally.

But I was sure that size = b->end - b->last; the size was not 0 at that time. I gdb checked the scene.

I still believe this problem is related to n = ngx_writev(c, &header); returning -2.