Opened 4 weeks ago

Last modified 4 weeks ago

#2658 new defect

proxy_set_body

Reported by: amolnar@… Owned by:
Priority: minor Milestone:
Component: nginx-module Version: 1.25.x
Keywords: proxy_set_body Cc: amolnar@…
uname -a: Linux fh1-lb01 5.15.0-1064-azure #73~20.04.1-Ubuntu SMP Mon May 6 09:43:44 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.25.1 (nginx-plus-r30)
built by gcc 9.3.0 (Ubuntu 9.3.0-10ubuntu2)
built with OpenSSL 1.1.1f 31 Mar 2020
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --build=nginx-plus-r30 --with-http_auth_jwt_module --with-http_f4f_module --with-http_hls_module --with-http_proxy_protocol_vendor_module --with-http_session_log_module --with-stream_mqtt_filter_module --with-stream_mqtt_preread_module --with-stream_proxy_protocol_vendor_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-plus-1.25.1/debian/debuild-base/nginx-plus-1.25.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

Description

We've been using NGINX Plus successfully for years at our company. We've recently enabled Cloudflare proxy https://developers.cloudflare.com/dns/manage-dns-records/reference/proxied-dns-records/ on our domains and noticed a bug/race condition possibly within NGINX's proxy_set_body.

We're using the ngx_http_realip_module https://nginx.org/en/docs/http/ngx_http_realip_module.html to set the x-real-ip header within NGINX for requests coming via Cloudflare https://www.cloudflare.com/ips/ . The request coming from Cloudflare hits our NGINX proxy which then forwards the requests to an upstream containing K8S cluster worker nodes.

Without using the Cloudflare proxy, the below location block works as expected, reaching our upstream service with the correct body set with proxy_set_body. However, when enabling Cloudflare proxy, the request
times out after 1 minute with a 504 status code.

location ~ ^/cep/data-feed/normal-hints/(\d+) {
  include /etc/nginx/restrictaccess.conf;

  limit_req zone=cep_data_feed_normal_hints burst=10;
  
  set $k8s_service "cep-hub-notification-api";
  set $parameters "/graphql";
    
  proxy_method POST;  
  proxy_set_body '{ "query":" query userNotifiableHintsDataFeed { userNotifiableHints(userId: $1, feedType: NORMAL_HINT) {  hintId firstHint isPedigree hintType  givenNames surnames hintCount oldRelevance rating familyTreeId familyTreeRef sourceCountry sourceCategory familyTreeTitle dateCreated nodeId ahnenNumber rootNodeId hintReference imageReference hintPlace hintYear pedigreeRelevance weightedRelevance searchRecencyRelevance relevance } }" }';
  include /etc/nginx/services/fmp/conf/k8s-service.conf;

  add_header 'Access-Control-Allow-Credentials' 'true';
}

Whilst the Cloudflare proxy mode enabled, as a workaround we assign the body to a variable and call proxy_set_body with it, so it starts working:

  set $proxy_body '{ "query":" query userNotifiableHintsDataFeed { userNotifiableHints(userId: $1, feedType: TREE_HINT) {  hintId firstHint isPedigree hintType  givenNames surnames hintCount oldRelevance rating familyTreeId familyTreeRef sourceCountry sourceCategory familyTreeTitle dateCreated nodeId ahnenNumber rootNodeId hintReference imageReference hintPlace hintYear pedigreeRelevance weightedRelevance searchRecencyRelevance relevance } }" }'; 
  proxy_set_body $proxy_body;

Looking at our distributed tracing span (the target service is written in TypeScript using Apollo GraphQL), it's reporting the request failed after 10 seconds with:

event	exception
exception.message	request aborted
exception.stacktrace	
BadRequestError: request aborted
    at IncomingMessage.onAborted (/usr/src/app/node_modules/express/node_modules/raw-body/index.js:245:10)
    at /otel-auto-instrumentation-nodejs/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:55
    at AsyncLocalStorage.run (node:async_hooks:346:14)
    at AsyncLocalStorageContextManager.with (/otel-auto-instrumentation-nodejs/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)
    at IncomingMessage.contextWrapper (/otel-auto-instrumentation-nodejs/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:32)
    at IncomingMessage.clsBind (/usr/src/app/node_modules/cls-hooked/context.js:172:17)
    at IncomingMessage.emit (node:events:518:28)
    at IncomingMessage.emitted (/usr/src/app/node_modules/emitter-listener/listener.js:134:21)
    at IncomingMessage._destroy (node:_http_incoming:224:10)
    at _destroy (node:internal/streams/destroy:121:10)
exception.type	ECONNABORTED

Why would there be a discrepancy with the proxy_set_body directive when the contents are specified inline (without the extra variable) and the Cloudflare proxy disabled, but result in aborted connections when Cloudflare proxy is enabled?

We've updated our known proxy_set_body directives to use a variable with the desired body, but there is no documentation/issue describing the problem, this issue can come up in the future for us/others.

Change History (2)

comment:1 by Roman Arutyunyan, 4 weeks ago

Your proxy_set_body contains an unnamed regex capture $1. Normally there's no problem with this and the value should be taken from the location regex match. However if there's another regex match along the way, that match will interfere with the result. Try switching to a named capture ?<foo> instead.

comment:2 by amolnar@…, 4 weeks ago

Thank you Roman for the explanation. We will give named captures a go shortly.

Note: See TracTickets for help on using tickets.