Opened 4 years ago

Closed 4 years ago

Last modified 2 years ago

#2000 closed defect (invalid)

"upstream sent invalid chunked response while reading upstream" when responding a stream.

Reported by: basti.lamberto@… Owned by:
Priority: major Milestone:
Component: nginx-core Version: 1.19.x
Keywords: proxy Cc:
uname -a: Linux rpi2 5.4.0-1011-raspi #11-Ubuntu SMP Fri May 8 07:49:33 UTC 2020 armv7l armv7l armv7l GNU/Linux
nginx -V: nginx version: nginx/1.19.0
built by gcc 8.3.0 (Debian 8.3.0-6)
built with OpenSSL 1.1.1d 10 Sep 2019
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/tmp/tmp.EorXWWuU3Y/nginx-1.19.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

Description

A detailed log is available at https://github.com/ktorio/ktor/issues/1933

It seems that nginx closes the proxy way before my Ktor actually starts responding.

It only happens when responding using a stream, otherwise everything works.

Attachments (1)

nginx.log.7z (18.7 KB ) - added by basti.lamberto@… 4 years ago.

Download all attachments as: .zip

Change History (8)

comment:1 by Maxim Dounin, 4 years ago

I don't see any nginx-related details in the issue linked except the error message itself, "upstream sent invalid chunked response while reading upstream". The message suggests that backend returned a broken response, with invalid chunked encoding. That is, your backend started returning a response, but the response contained invalid chunked encoding framing. Details, if needed, can be found in nginx debugging log.

Could you please elaborate what makes you think this is a bug in nginx rather than a broken response from backend server, as the message suggests?

by basti.lamberto@…, 4 years ago

Attachment: nginx.log.7z added

comment:2 by basti.lamberto@…, 4 years ago

Hi, Thank you for your time. Indeed I am very new into web protocols and I am practically flying blind. The endpoint works when exposing it directly.

I've attached the log! It seems that somehow the client closes the connection. But why?

Version 0, edited 4 years ago by basti.lamberto@… (next)

comment:3 by Maxim Dounin, 4 years ago

Resolution: invalid
Status: newclosed

Relevant part from the log:

2020/06/08 20:29:15 [debug] 29#29: *4 http upstream request: "/sql/wordClouds/tweets/hope?"
2020/06/08 20:29:15 [debug] 29#29: *4 http upstream process header
2020/06/08 20:29:15 [debug] 29#29: *4 malloc: 0172C040:4096
2020/06/08 20:29:15 [debug] 29#29: *4 recv: eof:0, avail:-1
2020/06/08 20:29:15 [debug] 29#29: *4 recv: fd:3 154 of 4096
2020/06/08 20:29:15 [debug] 29#29: *4 http proxy status 200 "200 "
2020/06/08 20:29:15 [debug] 29#29: *4 http proxy header: "Cache-Control: max-age=600"
2020/06/08 20:29:15 [debug] 29#29: *4 http proxy header: "Transfer-Encoding: chunked"
2020/06/08 20:29:15 [debug] 29#29: *4 http proxy header: "Connection: close"
2020/06/08 20:29:15 [debug] 29#29: *4 http proxy header: "Content-Type: image/png"
2020/06/08 20:29:15 [debug] 29#29: *4 http proxy header: "Date: Mon, 08 Jun 2020 20:29:15 GMT"
2020/06/08 20:29:15 [debug] 29#29: *4 http proxy header done
2020/06/08 20:29:15 [debug] 29#29: *4 HTTP/1.1 200 
Server: nginx/1.19.0
Date: Mon, 08 Jun 2020 20:29:15 GMT
Content-Type: image/png
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: max-age=600

2020/06/08 20:29:15 [debug] 29#29: *4 write new buf t:1 f:0 0172E320, pos 0172E320, size: 181 file: 0, size: 0
2020/06/08 20:29:15 [debug] 29#29: *4 http write filter: l:0 f:0 s:181
2020/06/08 20:29:15 [debug] 29#29: *4 http cacheable: 0
2020/06/08 20:29:15 [debug] 29#29: *4 http proxy filter init s:200 h:0 c:1 l:-1
2020/06/08 20:29:15 [debug] 29#29: *4 http upstream process upstream
2020/06/08 20:29:15 [debug] 29#29: *4 pipe read upstream: 0
2020/06/08 20:29:15 [debug] 29#29: *4 pipe preread: 0
2020/06/08 20:29:15 [debug] 29#29: *4 pipe buf free s:0 t:1 f:0 0172C040, pos 0172C0DA, size: 0 file: 0, size: 0
2020/06/08 20:29:15 [debug] 29#29: *4 pipe length: 3
2020/06/08 20:29:15 [debug] 29#29: *4 event timer del: 3: 573678
2020/06/08 20:29:15 [debug] 29#29: *4 event timer add: 3: 60000:575076
2020/06/08 20:29:15 [debug] 29#29: *4 http upstream request: "/sql/wordClouds/tweets/hope?"
2020/06/08 20:29:15 [debug] 29#29: *4 http upstream dummy handler
2020/06/08 20:29:15 [debug] 29#29: *4 http upstream request: "/sql/wordClouds/tweets/hope?"
2020/06/08 20:29:15 [debug] 29#29: *4 http upstream process upstream
2020/06/08 20:29:15 [debug] 29#29: *4 pipe read upstream: 1
2020/06/08 20:29:15 [debug] 29#29: *4 readv: eof:0, avail:-1
2020/06/08 20:29:15 [debug] 29#29: *4 readv: 1, last:3942
2020/06/08 20:29:15 [debug] 29#29: *4 pipe recv chain: 33
2020/06/08 20:29:15 [debug] 29#29: *4 pipe buf free s:0 t:1 f:0 0172C040, pos 0172C0DA, size: 33 file: 0, size: 0
2020/06/08 20:29:15 [debug] 29#29: *4 pipe length: 3
2020/06/08 20:29:15 [debug] 29#29: *4 http chunked byte: 89 s:0
2020/06/08 20:29:15 [error] 29#29: *4 upstream sent invalid chunked response while reading upstream, client: 10.0.0.2, server: , request: "GET /sql/wordClouds/tweets/hope HTTP/1.1", upstream: "http://10.0.2.5:80/wordClouds/tweets/hope", host: "192.168.1.158"

So the upstream server returns a response with "Transfer-Encoding: chunked", and the first byte of the response body is 0x89. This character is outside of the ASCII range and clearly is not a hexadecimal number as required by the chunked encoding.

Clearly this is an incorrect response from the backend. Depending on how the framework you are using is expected to work, this may either indicate a bug in the framework, or you are using the framework incorrectly.

comment:4 by basti.lamberto@…, 4 years ago

Thank you very much :) I've updated the issue on the Ktor board. Just one question tho, why Chrome and other clients correctly handles the request when not proxied?

comment:5 by Maxim Dounin, 4 years ago

Just one question tho, why Chrome and other clients correctly handles the request when not proxied?

My best guess that's because Chrome is using HTTP/2 when talking to your backend directly, and the backend works correctly via HTTP/2.

comment:6 by lamba92@…, 4 years ago

I was using the IntelliJ HTTP tool and noticed that the request is served using HTTP/1.1:

GET http://localhost/wordClouds/tweets/anger

HTTP/1.1 200 
Cache-Control: max-age=600
Transfer-Encoding: chunked
Connection: keep-alive, keep-alive
Content-Type: image/png
Transfer-Encoding: chunked
Date: Sat, 13 Jun 2020 14:33:02 GMT
Keep-Alive: timeout=60

It actually works. Indeed reading the RFC you linked, it states:

Page 19

A recipient MUST parse an HTTP message as a sequence of octets in an
   encoding that is a superset of US-ASCII [USASCII].

0x89 is indeed outside the range of ASCII but what a superset of US-ASCII means? And why 0x89 is not in the range of a superset?

comment:7 by Maxim Dounin, 4 years ago

The first byte of the message body encoded with chunked encoding must be a HEXDIG ("hexadecimal 0-9/A-F/a-f"), and a character outside of the ASCII range certainly doesn't fit. See above for the normative reference.

Note: See TracTickets for help on using tickets.