Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#950 closed defect (fixed)

Bad gateway with http2 requests on hhvm fastcgi since 1.9.14

Reported by: GregOriol@… Owned by:
Priority: major Milestone:
Component: nginx-module Version: 1.9.x
Keywords: Cc:
uname -a: Linux server 3.14.32-xxxx-std-ipv6-64 #1 SMP Sat Feb 7 12:07:25 CET 2015 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.9.14
built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.1)
built with OpenSSL 1.0.1f 6 Jan 2014
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_perl_module=dynamic --add-dynamic-module=debian/extra/njs-91543c86f412/nginx --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-mail --with-mail_ssl_module --with-file-aio --with-http_v2_module --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--as-needed' --with-ipv6

Description

Since nginx 1.9.14, using an http2 server with hhvm as fastcgi backend doesn't seem to work: nginx returns a 502 Bad Gateway.

In the errorlog, there is this message:

2016/04/10 13:36:34 [error] 9763#9763: *7 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: [IP], server: [SERVER_NAME], request: "GET /test.php HTTP/2.0", upstream: "fastcgi://127.0.0.1:9000", host: "[SERVER_NAME]"

Nothing special on hhvm's side; the php code used in test.php is a simple <?php echo "ok".

Works well with nginx 1.9.13
Works well with php-fpm as fastcgi backend
Works well with http/1.1 requests; only http2 requests are having this 502 error

Change History (7)

comment:1 by Norbert Fischer, 4 years ago

That looks similar to the errors I got after upgrading to 1.9.14 today.

I upgraded from

1.9.13-1~jessie

to

1.9.14-1~jessie

Error message:

2016/04/10 10:23:20 [error] 5607#5607: *5 readv() failed (104: Connection reset by peer) while reading upstream, client: [IP_ADDRESS], server: [SERVER_NAME] request: "GET /favicon.ico HTTP/2.0", upstream: "fastcgi://unix:/tmp/php-fastcgi.socket:", host: "[HOST_NAME]"

Observations:

  • Pages did indeed load but only partly
  • Disabling HTTP/2 in my browser did help
  • Disabling HTTP/2 in nginx would probably help too but I didn't try that

So it looks like when using HTTP/2 nginx tries to multiplex/pipeline connections to the fastcgi backend which refuses to comply.

Workaround:
In the end I added

fastcgi_keep_conn on;

to my php location which seems to help but I'm unsure if that's enough or if there may be other issues I'm not aware of so far.

comment:2 by Valentin V. Bartenev, 4 years ago

Please try the following patch:

diff -r 062f70adeaca -r 3d3f8c64d442 src/http/modules/ngx_http_fastcgi_module.c
--- a/src/http/modules/ngx_http_fastcgi_module.c        Mon Apr 11 18:02:11 2016 +0300
+++ b/src/http/modules/ngx_http_fastcgi_module.c        Mon Apr 11 18:42:34 2016 +0300
@@ -1177,6 +1177,11 @@ ngx_http_fastcgi_create_request(ngx_http
 
         while (body) {
 
+            if (ngx_buf_special(body->buf)) {
+                body = body->next;
+                continue;
+            }
+
             if (body->buf->in_file) {
                 file_pos = body->buf->file_pos;
 

in reply to:  2 comment:3 by Norbert Fischer, 4 years ago

Replying to vbart:

Please try the following patch:

diff -r 062f70adeaca -r 3d3f8c64d442 src/http/modules/ngx_http_fastcgi_module.c
--- a/src/http/modules/ngx_http_fastcgi_module.c        Mon Apr 11 18:02:11 2016 +0300
+++ b/src/http/modules/ngx_http_fastcgi_module.c        Mon Apr 11 18:42:34 2016 +0300
@@ -1177,6 +1177,11 @@ ngx_http_fastcgi_create_request(ngx_http
 
         while (body) {
 
+            if (ngx_buf_special(body->buf)) {
+                body = body->next;
+                continue;
+            }
+
             if (body->buf->in_file) {
                 file_pos = body->buf->file_pos;
 

That works for me. After applying the patch, pages load completely and no more error messages come up.

comment:4 by Valentin Bartenev <vbart@…>, 4 years ago

In 6512:b5734248d5e7/nginx:

FastCGI: skip special bufs in buffered request body chain.

This prevents forming empty records out of such buffers. Particularly it fixes
double end-of-stream records with chunked transfer encoding, or when HTTP/2 is
used and the END_STREAM flag has been sent without data. In both cases there
is an empty buffer at the end of the request body chain with the "last_buf"
flag set.

The canonical libfcgi, as well as php implementation, tolerates such records,
while the HHVM parser is more strict and drops the connection (ticket #950).

comment:5 by Ruslan Ermilov, 4 years ago

Resolution: fixed
Status: newclosed

Fixed, thanks for testing!

comment:6 by raeesiqbal@…, 4 years ago

When will this fix be added to mainstream nginx?

comment:7 by Maxim Dounin, 4 years ago

The fix is committed 4 weeks ago as revision b5734248d5e7 (see comment:4), and it is available as part of nginx 1.9.15 released at 19 Apr 2016:

    *) Bugfix: "recv() failed" errors might occur when using HHVM as a
       FastCGI server.
Note: See TracTickets for help on using tickets.