id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc	uname	nginx_version
2457	Nginx changes HTTP 413 Payload Too Large response status to HTTP 502 Bad Gateway response status	Tobias Riemenschneider		"I am running a Vert.x core HTTP server behind a Nginx load balancer and limiting the body size via a `BodyHandler` in the Vert.x core HTTP server.
When hitting the body limit, the Vert.x core HTTP responds with HTTP `413 Payload Too Large` response status. In most of the cases, Nginx simply passes this response to the client. But sometimes Nginx responds with HTTP `502 Bad Gateway` response status.

After enabling request and network activity logging and checking the Nginx error log I found that in these cases Nginx changes HTTP `413 Payload Too Large` response status to HTTP `502 Bad Gateway` response status due to an error
{{{
2022/10/28 21:08:12 [error] 9072#9072: *3583678 upstream prematurely closed connection while reading response header from upstream, client: 172.156.0.99, server: management.datahub5.stage.c8y.io, request: ""POST /service/datahub/scheduler/gettriggerstatus HTTP/1.1"", upstream: ""http://172.156.0.197:8303/service/datahub/scheduler/gettriggerstatus"", host: ""cdh-testing2.datahub5.stage.c8y.io:443""
}}}

I had a look at the implementation of `BodyHandlerImpl` in Vert.x and it looks like the `413 Payload Too Large` response status is already sent when the request contains HTTP `Content-Length` header and the set value is larger than the configured body limit.
Nginx is able to handle that situation most of the time but sometimes the mentioned error is logged and the response status is changes as described. It looks like Nginx is still reading the response while the Vert.x core HTTP server already closed the connection.

I created a reproducer for that issue. You can find it in the public GitHub repository riemenschneider/VertxWebIssue2295. Just clone the repo and run `gradlew cleanTest test`. The test uses port 9090 for the Vert.x HttpServer and port 9091 for the Nginx loadbalancer. If you want to run the test using other ports you can configure it via system properties (`gradlew cleanTest test -DHTTP_SERVER_PORT=9096 -DLOADBALANCER_PORT=9097 -DLOG_ACTIVITY=true`). Beware of turning log activity on since I had to play around with 1MB payloads! (For more details see README.md of the repository containing the reproducer.)
The Nginx loadbalancer is running in Docker and is configured via 
{{{
events { }

http {
  server {
    listen 0.0.0.0:9091;
    
    location / {
      proxy_pass http://backend;
    }
  }
  
  upstream backend {
    server host.docker.internal:9090;
  }
  
  client_max_body_size 10M;
}
}}}

Interestingly, I was not able to deterministically reproduce the issue in this reproducer although we can reproduce it deterministically in our product test suite. Therefore, I repeat the test several times which leads to a reproduction in nearly every test run."	defect	closed	major		nginx-core	1.23.x	duplicate			Linux 38dbd7b91b9f 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 GNU/Linux	"nginx version: nginx/1.23.3
built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
built with OpenSSL 1.1.1n  15 Mar 2022
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -ffile-prefix-map=/data/builder/debuild/nginx-1.23.3/debian/debuild-base/nginx-1.23.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'"