Opened 6 months ago
Last modified 6 months ago
#2658 new defect
proxy_set_body
| Reported by: | Owned by: | ||
|---|---|---|---|
| Priority: | minor | Milestone: | |
| Component: | nginx-module | Version: | 1.25.x |
| Keywords: | proxy_set_body | Cc: | amolnar@… |
| uname -a: | Linux fh1-lb01 5.15.0-1064-azure #73~20.04.1-Ubuntu SMP Mon May 6 09:43:44 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | ||
| nginx -V: |
nginx version: nginx/1.25.1 (nginx-plus-r30)
built by gcc 9.3.0 (Ubuntu 9.3.0-10ubuntu2) built with OpenSSL 1.1.1f 31 Mar 2020 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --build=nginx-plus-r30 --with-http_auth_jwt_module --with-http_f4f_module --with-http_hls_module --with-http_proxy_protocol_vendor_module --with-http_session_log_module --with-stream_mqtt_filter_module --with-stream_mqtt_preread_module --with-stream_proxy_protocol_vendor_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-plus-1.25.1/debian/debuild-base/nginx-plus-1.25.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie' |
||
Description
We've been using NGINX Plus successfully for years at our company. We've recently enabled Cloudflare proxy https://developers.cloudflare.com/dns/manage-dns-records/reference/proxied-dns-records/ on our domains and noticed a bug/race condition possibly within NGINX's proxy_set_body.
We're using the ngx_http_realip_module https://nginx.org/en/docs/http/ngx_http_realip_module.html to set the x-real-ip header within NGINX for requests coming via Cloudflare https://www.cloudflare.com/ips/ . The request coming from Cloudflare hits our NGINX proxy which then forwards the requests to an upstream containing K8S cluster worker nodes.
Without using the Cloudflare proxy, the below location block works as expected, reaching our upstream service with the correct body set with proxy_set_body. However, when enabling Cloudflare proxy, the request
times out after 1 minute with a 504 status code.
location ~ ^/cep/data-feed/normal-hints/(\d+) {
include /etc/nginx/restrictaccess.conf;
limit_req zone=cep_data_feed_normal_hints burst=10;
set $k8s_service "cep-hub-notification-api";
set $parameters "/graphql";
proxy_method POST;
proxy_set_body '{ "query":" query userNotifiableHintsDataFeed { userNotifiableHints(userId: $1, feedType: NORMAL_HINT) { hintId firstHint isPedigree hintType givenNames surnames hintCount oldRelevance rating familyTreeId familyTreeRef sourceCountry sourceCategory familyTreeTitle dateCreated nodeId ahnenNumber rootNodeId hintReference imageReference hintPlace hintYear pedigreeRelevance weightedRelevance searchRecencyRelevance relevance } }" }';
include /etc/nginx/services/fmp/conf/k8s-service.conf;
add_header 'Access-Control-Allow-Credentials' 'true';
}
Whilst the Cloudflare proxy mode enabled, as a workaround we assign the body to a variable and call proxy_set_body with it, so it starts working:
set $proxy_body '{ "query":" query userNotifiableHintsDataFeed { userNotifiableHints(userId: $1, feedType: TREE_HINT) { hintId firstHint isPedigree hintType givenNames surnames hintCount oldRelevance rating familyTreeId familyTreeRef sourceCountry sourceCategory familyTreeTitle dateCreated nodeId ahnenNumber rootNodeId hintReference imageReference hintPlace hintYear pedigreeRelevance weightedRelevance searchRecencyRelevance relevance } }" }';
proxy_set_body $proxy_body;
Looking at our distributed tracing span (the target service is written in TypeScript using Apollo GraphQL), it's reporting the request failed after 10 seconds with:
event exception
exception.message request aborted
exception.stacktrace
BadRequestError: request aborted
at IncomingMessage.onAborted (/usr/src/app/node_modules/express/node_modules/raw-body/index.js:245:10)
at /otel-auto-instrumentation-nodejs/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:55
at AsyncLocalStorage.run (node:async_hooks:346:14)
at AsyncLocalStorageContextManager.with (/otel-auto-instrumentation-nodejs/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)
at IncomingMessage.contextWrapper (/otel-auto-instrumentation-nodejs/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:32)
at IncomingMessage.clsBind (/usr/src/app/node_modules/cls-hooked/context.js:172:17)
at IncomingMessage.emit (node:events:518:28)
at IncomingMessage.emitted (/usr/src/app/node_modules/emitter-listener/listener.js:134:21)
at IncomingMessage._destroy (node:_http_incoming:224:10)
at _destroy (node:internal/streams/destroy:121:10)
exception.type ECONNABORTED
Why would there be a discrepancy with the proxy_set_body directive when the contents are specified inline (without the extra variable) and the Cloudflare proxy disabled, but result in aborted connections when Cloudflare proxy is enabled?
We've updated our known proxy_set_body directives to use a variable with the desired body, but there is no documentation/issue describing the problem, this issue can come up in the future for us/others.
Change History (2)
comment:1 by , 6 months ago
comment:2 by , 6 months ago
Thank you Roman for the explanation. We will give named captures a go shortly.
Your
proxy_set_bodycontains an unnamed regex capture$1. Normally there's no problem with this and the value should be taken from the location regex match. However if there's another regex match along the way, that match will interfere with the result. Try switching to a named capture?<foo>instead.