id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc	uname	nginx_version
2376	GRPC: upstream rejected request with error while reading response header from upstream	erebe@…		"Hello,

Reporting an issue with GRPC where nginx is not correctly forwarding to client the response and RST_STREAM
when server early abort the http2 stream.

My setup:

[client] === grpc stream (with ssl) ===> [nginx] === grpc stream (cleartext) ===> [backend]


Description:

When the client is establishing a grpc (client/unidirectional) stream, it may happen that the backend close/reset the http2 stream before exhausting the client input stream, for example due to a timeout.

In this case, the backend send back the response and reset the stream:
{{{

grpc-gateway DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(1), flags: (0x4: END_HEADERS) }
grpc-gateway DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Data { stream_id: StreamId(1) }
grpc-gateway DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(1), flags: (0x5: END_HEADERS | END_STREAM) }
grpc-gateway DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Reset { stream_id: StreamId(1), error_code: CANCEL }
}}}


My issue is that the client never receive the response nor the http2 RST_STREAM from nginx.
The client discover that the stream have been reset only after the next client side http2 PING frame, or after the client try to send some new data. 

{{{
DEBUG Connection{peer=Client}: h2::codec::framed_write: send frame=Ping { ack: false, payload: [59, 124, 219, 122, 11, 135, 22, 180] }
DEBUG status_request{request_id=117db9f2-707c-4ccd-a824-f0f8d36c9fa3}:Connection{peer=Client}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(1), flags: (0x4: END_HEADERS) }
DEBUG status_request{request_id=117db9f2-707c-4ccd-a824-f0f8d36c9fa3}:Connection{peer=Client}: h2::codec::framed_read: received frame=Reset { stream_id: StreamId(1), error_code: INTERNAL_ERROR }
DEBUG hyper::proto::h2::client: client request body error: error writing a body to connection: send stream capacity unexpectedly closed
}}}


The only thing I see in the nginx log is those messages:
(error 8 is http2 CANCEL error code)

{{{
[error] 855#855: *73638 upstream rejected request with error 8 while reading response header from upstream, client: 10.0.33.7, server: grpc.qovery.com, request: ""POST /agent.Agent/AgentResponsePublish HTTP/2.0"", upstream: ""grpc://10.0.20.240:8081"", host: ""grpc.qovery.com:443""
10.0.33.7 - - [10/Aug/2022:21:48:29 +0000] ""POST /agent.Agent/AgentResponsePublish HTTP/2.0"" 200 0 ""-"" ""tonic/0.8.0"" 566 60.024 [qovery-prod-grpc-gateway-grpc] [] 10.0.20.240:8081 68 60.025 200 e0ee79d5fd24a5bbf39556e159cb3840  
}}}


If I remove nginx from my setup everything is working as expected.

The issue seems a duplicate of this one https://trac.nginx.org/nginx/ticket/1792, but as the ticket has been closed, I opened a new one with detailed information.
 

"	defect	reopened	minor		nginx-module	1.19.x				"Linux nginx-grpc-ingress-ingress-nginx-nginx-grpc-controller-7b6pvg7j 5.4.188-104.359.amzn2.x86_64 #1 SMP Thu Apr 14 20:53:13 UTC 2022 x86_64 Linux
"	"nginx version: nginx/1.19.10
built by gcc 11.2.1 20220219 (Alpine 11.2.1_git20220219)
built with OpenSSL 1.1.1q  5 Jul 2022
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx --conf-path=/etc/nginx/nginx.conf --modules-path=/etc/nginx/modules --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-compat --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_geoip_module --with-http_gzip_static_module --with-http_sub_module --with-http_v2_module --with-stream --with-stream_ssl_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-http_secure_link_module --with-http_gunzip_module --with-file-aio --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module --without-http_uwsgi_module --without-http_scgi_module --with-cc-opt='-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wno-deprecated-declarations -fno-strict-aliasing -D_FORTIFY_SOURCE=2 --param=ssp-buffer-size=4 -DTCP_FASTOPEN=23 -fPIC -I/root/.hunter/_Base/d45d77d/d0c447a/3b7ee27/Install/include -Wno-cast-function-type -m64 -mtune=generic' --with-ld-opt='-fPIE -fPIC -pie -Wl,-z,relro -Wl,-z,now -L/root/.hunter/_Base/d45d77d/d0c447a/3b7ee27/Install/lib' --user=www-data --group=www-data --add-module=/tmp/build/ngx_devel_kit-0.3.1 --add-module=/tmp/build/set-misc-nginx-module-0.32 --add-module=/tmp/build/headers-more-nginx-module-0.33 --add-module=/tmp/build/ngx_http_substitutions_filter_module-b8a71eacc7f986ba091282ab8b1bbbc6ae1807e0 --add-module=/tmp/build/lua-nginx-module-b721656a9127255003b696b42ccc871c7ec18d59 --add-module=/tmp/build/stream-lua-nginx-module-74f8c8bca5b95cecbf42d4e1a465bc08cd075a9b --add-module=/tmp/build/lua-upstream-nginx-module-8aa93ead98ba2060d4efd594ae33a35d153589bf --add-module=/tmp/build/nginx_ajp_module-a964a0bcc6a9f2bfb82a13752d7794a36319ffac --add-dynamic-module=/tmp/build/nginx-http-auth-digest-1.0.0 --add-dynamic-module=/tmp/build/nginx-influxdb-module-5b09391cb7b9a889687c0aa67964c06a2d933e8b --add-dynamic-module=/tmp/build/nginx-opentracing-0.19.0/opentracing --add-dynamic-module=/tmp/build/ModSecurity-nginx-1.0.2 --add-dynamic-module=/tmp/build/ngx_http_geoip2_module-a26c6beed77e81553686852dceb6c7fdacc5970d --add-dynamic-module=/tmp/build/ngx_brotli"