Opened 7 months ago
#2548 new defect
Worker infinite loop in ngx_http_do_read_client_request_body()
| Reported by: | Owned by: | ||
|---|---|---|---|
| Priority: | minor | Milestone: | |
| Component: | documentation | Version: | 1.25.x |
| Keywords: | Cc: | anight@… | |
| uname -a: | Linux server 6.2.0-32-generic #32-Ubuntu SMP PREEMPT_DYNAMIC Mon Aug 14 10:03:50 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | ||
| nginx -V: |
nginx version: nginx/1.25.2
built by gcc 12.3.0 (Ubuntu 12.3.0-1ubuntu1~23.04) built with OpenSSL 3.0.8 7 Feb 2023 TLS SNI support enabled configure arguments: --with-debug --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g3 -O0' |
||
Description
Consider the following location configuration:
location = @grpcweb {
rewrite ^ $orig_grpc_uri break;
proxy_pass http://127.0.0.1:82;
proxy_redirect off;
proxy_buffering off;
proxy_request_buffering off;
allow all;
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header X-Forwarded-Proto $scheme;
client_body_buffer_size 0;
client_max_body_size 0;
proxy_max_temp_file_size 0;
proxy_read_timeout 18000;
proxy_send_timeout 18000;
gzip off;
}
I figured out that in my case ngx_http_do_read_client_request_body() is called with zero sized buffer (rb->buf->start == rb->buf->end) and as a result it loops endlessly logging "http client request body rest 3" message thousands times a second.
The zero-sized buffer is created earlier in ngx_http_read_client_request_body():
/* TODO: honor r->request_body_in_single_buf */
if (!r->headers_in.chunked && rb->rest < size) {
size = (ssize_t) rb->rest;
if (r->request_body_in_single_buf) {
size += preread;
}
if (size == 0) {
size++;
}
} else {
size = clcf->client_body_buffer_size; // zero-sized buffer created here
}
rb->buf = ngx_create_temp_buf(r->pool, size);
if (rb->buf == NULL) {
rc = NGX_HTTP_INTERNAL_SERVER_ERROR;
goto done;
}
r->read_event_handler = ngx_http_read_client_request_body_handler;
r->write_event_handler = ngx_http_request_empty_handler;
rc = ngx_http_do_read_client_request_body(r);
So as I understand in my case the whole request didn't fit r->headers_in by 3 bytes and as result rb->rest was 3.
When I removed "client_body_buffer_size 0;" from configuration the bug disappeared.
I blindly copied the set of proxied location directives from some page on stackoverflow and didn't realize "client_body_buffer_size 0;" would cause endless loop in a worker process.
