Chunked FastCGI output gets chunked an extra time

[jmarshall.com@…]

When my FastCGI script returns a chunked body, with "Transfer-Encoding: chunked", nginx applies an extra layer of chunking to it. In other words, it takes the entire chunked output and wraps *that* in a single large chunk. For example, the first (FastCGI) response below results in the second (nginx) response:

================================
Status: 200 OK
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: close

5
hello
6
_world
0

================================
HTTP/1.1 200 OK
Server: nginx/1.13.1
Date: Mon, 12 Jun 2017 02:13:57 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: close

14
5
hello
6
_world
0

0

================================

I'm attaching a short Perl FastCGI script that demonstrates this bug. Here's the block for it in nginx.conf:

location /test/ {

fastcgi_pass 127.0.0.1:8003;

}

I've looked through old bugs, and this bug is alluded to several times, but those tickets have been closed. However, I just upgraded nginx from 1.4.7 to 1.31.1 and the problem happens in both.

Marking this as "major" since chunked encoding is critical for serving long script output in reasonable time, and this bug essentially prevents chunked encoding from being used.

[jmarshall.com@…]

Demonstrates the bug with FastCGI and chunked encoding

[Valentin V. Bartenev]

There's no such thing as HTTP chunked transfer encoding in FastCGI protocol. "Transfer-Encoding" is a hop-by-hop HTTP header which is only meaningful for two endpoints talking over HTTP/1.1 protocol, while your application talks FastCGI.

Also note that actually every FastCGI transfer already consists of chunks, it's encoded in one or more FastCGI records.

[jmarshall.com@…]

But nginx modifies the chunked HTTP response coming from the FastCGI script, and gets it wrong. The two "endpoints" in this case are the FastCGI script (which outputs HTTP over FastCGI) and the HTTP client, with nginx in the middle. It's the HTTP layer I'm talking about.

The short script I attached demonstrates the bug very clearly. But you don't even have to run it-- just look at the sample HTTP response I posted, as it gets changed from the script output into the nginx output. Do you actually think this response from the FastCGI script should result in this response from nginx?

This bug has been reported a few times, and each time was dismissed because of a reason like "there's no such thing as HTTP chunked transfer encoding in FastCGI protocol." But that ignores the very real problem that nginx has with chunked encoding in the HTTP protocol.

I want to be able to recommend nginx to my users (and I've modified my software significantly to support nginx), but as long as chunked encoding isn't handled correctly, I can't recommend it. :(

[Valentin V. Bartenev]

But nginx modifies the chunked HTTP response coming from the FastCGI script, and gets it wrong. The two "endpoints" in this case are the FastCGI script (which outputs HTTP over FastCGI) and the HTTP client, with nginx in the middle. It's the HTTP layer I'm talking about

FastCGI isn't based on HTTP, but on CGI. According to the ​CGI specification:

The script MUST NOT return any header fields that relate to
client-side communication issues and could affect the server's
ability to send the response to the client. The server MAY remove
any such header fields returned by the client. It SHOULD resolve any
conflicts between header fields returned by the script and header
fields that it would otherwise send itself.

That's it.

The short script I attached demonstrates the bug very clearly. But you don't even have to run it-- just look at the sample HTTP response I posted, as it gets changed from the script output into the nginx output. Do you actually think this response from the FastCGI script should result in this response from nginx?

Yes, nginx behaves in line with the CGI specification. It returns exactly what your scripts outputs with addition of HTTP transfer encoding over it.

This bug has been reported a few times, and each time was dismissed because of a reason like "there's no such thing as HTTP chunked transfer encoding in FastCGI protocol." But that ignores the very real problem that nginx has with chunked encoding in the HTTP protocol.

The problem is in the script that violates the specification, not in nginx. Please, fix your script.