id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc	uname	nginx_version
2658	proxy_set_body	amolnar@…		"We've been using NGINX Plus successfully for years at our company. We've recently enabled Cloudflare proxy https://developers.cloudflare.com/dns/manage-dns-records/reference/proxied-dns-records/ on our domains and noticed a bug/race condition possibly within NGINX's `proxy_set_body`.

We're using the `ngx_http_realip_module` https://nginx.org/en/docs/http/ngx_http_realip_module.html to set the `x-real-ip` header within NGINX for requests coming via Cloudflare https://www.cloudflare.com/ips/ . The request coming from Cloudflare hits our NGINX proxy which then forwards the requests to an upstream containing K8S cluster worker nodes.

Without using the Cloudflare proxy, the below `location` block works as expected, reaching our upstream service with the correct body set with `proxy_set_body`. However, when enabling Cloudflare proxy, the request
times out after 1 minute with a `504` status code.

{{{
location ~ ^/cep/data-feed/normal-hints/(\d+) {
  include /etc/nginx/restrictaccess.conf;

  limit_req zone=cep_data_feed_normal_hints burst=10;
  
  set $k8s_service ""cep-hub-notification-api"";
  set $parameters ""/graphql"";
    
  proxy_method POST;  
  proxy_set_body '{ ""query"":"" query userNotifiableHintsDataFeed { userNotifiableHints(userId: $1, feedType: NORMAL_HINT) {  hintId firstHint isPedigree hintType  givenNames surnames hintCount oldRelevance rating familyTreeId familyTreeRef sourceCountry sourceCategory familyTreeTitle dateCreated nodeId ahnenNumber rootNodeId hintReference imageReference hintPlace hintYear pedigreeRelevance weightedRelevance searchRecencyRelevance relevance } }"" }';
  include /etc/nginx/services/fmp/conf/k8s-service.conf;

  add_header 'Access-Control-Allow-Credentials' 'true';
}
}}}

Whilst the Cloudflare proxy mode enabled, as a workaround we assign the body to a variable and call `proxy_set_body` with it, so it starts working:

{{{
  set $proxy_body '{ ""query"":"" query userNotifiableHintsDataFeed { userNotifiableHints(userId: $1, feedType: TREE_HINT) {  hintId firstHint isPedigree hintType  givenNames surnames hintCount oldRelevance rating familyTreeId familyTreeRef sourceCountry sourceCategory familyTreeTitle dateCreated nodeId ahnenNumber rootNodeId hintReference imageReference hintPlace hintYear pedigreeRelevance weightedRelevance searchRecencyRelevance relevance } }"" }'; 
  proxy_set_body $proxy_body;
}}}

Looking at our distributed tracing span (the target service is written in TypeScript using Apollo GraphQL), it's reporting the request failed after 10 seconds with:
{{{
event	exception
exception.message	request aborted
exception.stacktrace	
BadRequestError: request aborted
    at IncomingMessage.onAborted (/usr/src/app/node_modules/express/node_modules/raw-body/index.js:245:10)
    at /otel-auto-instrumentation-nodejs/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:55
    at AsyncLocalStorage.run (node:async_hooks:346:14)
    at AsyncLocalStorageContextManager.with (/otel-auto-instrumentation-nodejs/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)
    at IncomingMessage.contextWrapper (/otel-auto-instrumentation-nodejs/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:32)
    at IncomingMessage.clsBind (/usr/src/app/node_modules/cls-hooked/context.js:172:17)
    at IncomingMessage.emit (node:events:518:28)
    at IncomingMessage.emitted (/usr/src/app/node_modules/emitter-listener/listener.js:134:21)
    at IncomingMessage._destroy (node:_http_incoming:224:10)
    at _destroy (node:internal/streams/destroy:121:10)
exception.type	ECONNABORTED
}}}

Why would there be a discrepancy with the `proxy_set_body` directive when the contents are specified inline (without the extra variable) and the Cloudflare proxy disabled, but result in aborted connections when Cloudflare proxy is enabled?

We've updated our known `proxy_set_body` directives to use a variable with the desired body, but there is no documentation/issue describing the problem, this issue can come up in the future for us/others."	defect	new	minor		nginx-module	1.25.x		proxy_set_body	amolnar@…	Linux fh1-lb01 5.15.0-1064-azure #73~20.04.1-Ubuntu SMP Mon May 6 09:43:44 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	"nginx version: nginx/1.25.1 (nginx-plus-r30)
built by gcc 9.3.0 (Ubuntu 9.3.0-10ubuntu2) 
built with OpenSSL 1.1.1f  31 Mar 2020
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --build=nginx-plus-r30 --with-http_auth_jwt_module --with-http_f4f_module --with-http_hls_module --with-http_proxy_protocol_vendor_module --with-http_session_log_module --with-stream_mqtt_filter_module --with-stream_mqtt_preread_module --with-stream_proxy_protocol_vendor_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-plus-1.25.1/debian/debuild-base/nginx-plus-1.25.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'"