Bad gateway with http2 requests on hhvm fastcgi since 1.9.14

[GregOriol@…]

Since nginx 1.9.14, using an http2 server with hhvm as fastcgi backend doesn't seem to work: nginx returns a 502 Bad Gateway.

In the errorlog, there is this message:

2016/04/10 13:36:34 [error] 9763#9763: *7 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: [IP], server: [SERVER_NAME], request: "GET /test.php HTTP/2.0", upstream: "fastcgi://127.0.0.1:9000", host: "[SERVER_NAME]"

Nothing special on hhvm's side; the php code used in test.php is a simple <?php echo "ok".

Works well with nginx 1.9.13
Works well with php-fpm as fastcgi backend
Works well with http/1.1 requests; only http2 requests are having this 502 error

[Norbert Fischer]

That looks similar to the errors I got after upgrading to 1.9.14 today.

I upgraded from

1.9.13-1~jessie

to

1.9.14-1~jessie

Error message:

2016/04/10 10:23:20 [error] 5607#5607: *5 readv() failed (104: Connection reset by peer) while reading upstream, client: [IP_ADDRESS], server: [SERVER_NAME] request: "GET /favicon.ico HTTP/2.0", upstream: "fastcgi://unix:/tmp/php-fastcgi.socket:", host: "[HOST_NAME]"

Observations:

• Pages did indeed load but only partly
• Disabling HTTP/2 in my browser did help
• Disabling HTTP/2 in nginx would probably help too but I didn't try that

So it looks like when using HTTP/2 nginx tries to multiplex/pipeline connections to the fastcgi backend which refuses to comply.

Workaround:
In the end I added

fastcgi_keep_conn on;

to my php location which seems to help but I'm unsure if that's enough or if there may be other issues I'm not aware of so far.

[Valentin V. Bartenev]

Please try the following patch:

diff -r 062f70adeaca -r 3d3f8c64d442 src/http/modules/ngx_http_fastcgi_module.c
--- a/src/http/modules/ngx_http_fastcgi_module.c        Mon Apr 11 18:02:11 2016 +0300
+++ b/src/http/modules/ngx_http_fastcgi_module.c        Mon Apr 11 18:42:34 2016 +0300
@@ -1177,6 +1177,11 @@ ngx_http_fastcgi_create_request(ngx_http
 
         while (body) {
 
+            if (ngx_buf_special(body->buf)) {
+                body = body->next;
+                continue;
+            }
+
             if (body->buf->in_file) {
                 file_pos = body->buf->file_pos;
 

[Norbert Fischer]

Replying to vbart:

Please try the following patch:

diff -r 062f70adeaca -r 3d3f8c64d442 src/http/modules/ngx_http_fastcgi_module.c
--- a/src/http/modules/ngx_http_fastcgi_module.c        Mon Apr 11 18:02:11 2016 +0300
+++ b/src/http/modules/ngx_http_fastcgi_module.c        Mon Apr 11 18:42:34 2016 +0300
@@ -1177,6 +1177,11 @@ ngx_http_fastcgi_create_request(ngx_http
 
         while (body) {
 
+            if (ngx_buf_special(body->buf)) {
+                body = body->next;
+                continue;
+            }
+
             if (body->buf->in_file) {
                 file_pos = body->buf->file_pos;
 

That works for me. After applying the patch, pages load completely and no more error messages come up.

[Valentin Bartenev <vbart@…>]

In 6512:b5734248d5e7/nginx:

FastCGI: skip special bufs in buffered request body chain.

This prevents forming empty records out of such buffers. Particularly it fixes
double end-of-stream records with chunked transfer encoding, or when HTTP/2 is
used and the END_STREAM flag has been sent without data. In both cases there
is an empty buffer at the end of the request body chain with the "last_buf"
flag set.

The canonical libfcgi, as well as php implementation, tolerates such records,
while the HHVM parser is more strict and drops the connection (ticket #950).

[Ruslan Ermilov]

Fixed, thanks for testing!

[raeesiqbal@…]

When will this fix be added to mainstream nginx?

[Maxim Dounin]

The fix is committed 4 weeks ago as revision b5734248d5e7 (see comment:4), and it is available as part of ​nginx 1.9.15 released at 19 Apr 2016:

    *) Bugfix: "recv() failed" errors might occur when using HHVM as a
       FastCGI server.