Nginx changes HTTP 413 Payload Too Large response status to HTTP 502 Bad Gateway response status

[Tobias Riemenschneider]

I am running a Vert.x core HTTP server behind a Nginx load balancer and limiting the body size via a BodyHandler in the Vert.x core HTTP server.
When hitting the body limit, the Vert.x core HTTP responds with HTTP 413 Payload Too Large response status. In most of the cases, Nginx simply passes this response to the client. But sometimes Nginx responds with HTTP 502 Bad Gateway response status.

After enabling request and network activity logging and checking the Nginx error log I found that in these cases Nginx changes HTTP 413 Payload Too Large response status to HTTP 502 Bad Gateway response status due to an error

2022/10/28 21:08:12 [error] 9072#9072: *3583678 upstream prematurely closed connection while reading response header from upstream, client: 172.156.0.99, server: management.datahub5.stage.c8y.io, request: "POST /service/datahub/scheduler/gettriggerstatus HTTP/1.1", upstream: "http://172.156.0.197:8303/service/datahub/scheduler/gettriggerstatus", host: "cdh-testing2.datahub5.stage.c8y.io:443"

I had a look at the implementation of BodyHandlerImpl in Vert.x and it looks like the 413 Payload Too Large response status is already sent when the request contains HTTP Content-Length header and the set value is larger than the configured body limit.
Nginx is able to handle that situation most of the time but sometimes the mentioned error is logged and the response status is changes as described. It looks like Nginx is still reading the response while the Vert.x core HTTP server already closed the connection.

I created a reproducer for that issue. You can find it in the public GitHub repository riemenschneider/VertxWebIssue2295. Just clone the repo and run gradlew cleanTest test. The test uses port 9090 for the Vert.x HttpServer and port 9091 for the Nginx loadbalancer. If you want to run the test using other ports you can configure it via system properties (gradlew cleanTest test -DHTTP_SERVER_PORT=9096 -DLOADBALANCER_PORT=9097 -DLOG_ACTIVITY=true). Beware of turning log activity on since I had to play around with 1MB payloads! (For more details see README.md of the repository containing the reproducer.)
The Nginx loadbalancer is running in Docker and is configured via

events { }

http {
  server {
    listen 0.0.0.0:9091;
    
    location / {
      proxy_pass http://backend;
    }
  }
  
  upstream backend {
    server host.docker.internal:9090;
  }
  
  client_max_body_size 10M;
}

Interestingly, I was not able to deterministically reproduce the issue in this reproducer although we can reproduce it deterministically in our product test suite. Therefore, I repeat the test several times which leads to a reproduction in nearly every test run.

[Maxim Dounin]

This looks like a missing lingering close on the backend server side, so RST is sent to nginx if sending request body to the backend happens to hit the already closed socket (see #1673, #1211, #1037 for detailed explanations), and this prevents nginx from getting any response. Does your backend server implements lingering close?

[Maxim Dounin]

Feedback timeout. As suggested in comment:1, this looks like yet another backend which fails to implement lingering close. This needs to be fixed on the backend side. Closing this as a duplicate of #1037.