Opened 2 years ago

Closed 2 years ago

Last modified 2 years ago

#2328 closed defect (invalid)

ngx_http_uwsgi_module removes chunked-encoding, but doesn't modify Transfer-Encoding header

Reported by: dener.kup@… Owned by:
Priority: major Milestone:
Component: nginx-module Version:
Keywords: uwsgi Cc:
uname -a: Linux 446f4b0329e7 5.15.8-1-default #1 SMP Wed Dec 15 08:12:54 UTC 2021 (0530e5c) x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.21.6
built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
built with OpenSSL 1.1.1k 25 Mar 2021
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -ffile-prefix-map=/data/builder/debuild/nginx-1.21.6/debian/debuild-base/nginx-1.21.6=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

Description

It looks like that ngx_http_uwsgi_module removes chunked encoding from the given request, but doesn't remove 'chunked' word from Transfer-Encoding header.

Documentation for uwsgi_request_buffering says that chunked encoding is always eliminated before proxying to upstream:

When HTTP/1.1 chunked transfer encoding is used to send the original request body, the request body will be buffered regardless of the directive value.

Upstream receives both CONTENT_LENGTH and HTTP_TRANSFER_ENCODING with 'chunked' specifier inside. In a case of uwsgi + werkzeug/flask as upstream that leads to invalid werkzeug behaviour as it ignores Content-Length due to 4.4 of RFC2616:

Messages MUST NOT include both a Content-Length header field and a non-identity transfer-coding. If the message does include a non-identity transfer-coding, the Content-Length MUST be ignored.

Minimal docker-compose project that demonstrates the problem is attached. Following steps reproduce the problem:

  1. unpack project
  2. docker-compose up --build
  3. cat request | nc localhost 8080
  4. see headers and request body in service logs:
    app_1    | {'CONTENT_LENGTH': '6',  <--------
    
    app_1    |  'CONTENT_TYPE': 'text/plain',
    app_1    |  'DOCUMENT_ROOT': '/etc/nginx/html',
    app_1    |  'HTTP_ACCEPT': '*/*',
    app_1    |  'HTTP_CONTENT_TYPE': 'text/plain',
    app_1    |  'HTTP_HOST': 'localhost:8080',
    
    app_1    |  'HTTP_TRANSFER_ENCODING': 'chunked', <--------
    
    app_1    |  'PATH_INFO': '/',
    app_1    |  'QUERY_STRING': '',
    app_1    |  'REMOTE_ADDR': '127.0.0.1',
    app_1    |  'REMOTE_PORT': '40198',
    app_1    |  'REQUEST_METHOD': 'POST',
    app_1    |  'REQUEST_SCHEME': 'http',
    app_1    |  'REQUEST_URI': '/',
    app_1    |  'SERVER_NAME': '',
    app_1    |  'SERVER_PORT': '8080',
    app_1    |  'SERVER_PROTOCOL': 'HTTP/1.1',
    app_1    |  'uwsgi.node': b'denisnotebook',
    app_1    |  'uwsgi.version': b'2.0.20',
    app_1    |  'wsgi.errors': <_io.TextIOWrapper name=2 mode='w' encoding='ANSI_X3.4-1968'>,
    app_1    |  'wsgi.file_wrapper': <built-in function uwsgi_sendfile>,
    app_1    |  'wsgi.input': <uwsgi._Input object at 0x7f613291ea30>,
    app_1    |  'wsgi.multiprocess': False,
    app_1    |  'wsgi.multithread': False,
    app_1    |  'wsgi.run_once': False,
    app_1    |  'wsgi.url_scheme': 'http',
    app_1    |  'wsgi.version': (1, 0)}
    
    app_1    | wsgi.input b'foobar' <--------
    
  1. The screenshot shows the data that uwsgi receives. You can see there is no chunks lengths in request body. Data was captured by tcpdump and visualized via wireshark.

PS: there is no 1.21.6 in Version dropdown. So I leave it unset.

Attachments (2)

tcpdump_of_uwsgi.png (357.8 KB ) - added by dener.kup@… 2 years ago.
compose_project.tar (20.0 KB ) - added by dener.kup@… 2 years ago.

Download all attachments as: .zip

Change History (5)

by dener.kup@…, 2 years ago

Attachment: tcpdump_of_uwsgi.png added

by dener.kup@…, 2 years ago

Attachment: compose_project.tar added

in reply to:  description comment:1 by Maxim Dounin, 2 years ago

Resolution: invalid
Status: newclosed

It looks like that ngx_http_uwsgi_module removes chunked encoding from the given request, but doesn't remove 'chunked' word from Transfer-Encoding header.

The HTTP_* uwsgi parameters (AKA environment variables) reflect the request headers of the HTTP request received by nginx. It does contain all the headers received from the client, including hop-by-hop headers, such as Connection and Transfer-Encoding. These variables are not expected to be used by the application to parse the request body: they apply to the HTTP request from the client, not to the uwsgi request from nginx to the application.

To read the request body over uwsgi, the CONTENT_LENGTH variable should be used instead, much like with other CGI-like protocols. See also RFC 3875 section 4.2, Request Message-Body.

Upstream receives both CONTENT_LENGTH and HTTP_TRANSFER_ENCODING with 'chunked' specifier inside. In a case of uwsgi + werkzeug/flask as upstream that leads to invalid werkzeug behaviour as it ignores Content-Length due to 4.4 of RFC2616:

This looks like something to be fixed in werkzeug. The request body in uwsgi request does not use any transfer encodings and has nothing do with HTTP/1.1 and chunked transfer encoding, even if it is used between nginx and the client and/or indicated by the HTTP_* variables.

Just in case its not clear, an obvious workaround on nginx side would be to use something like:

uwsgi_param HTTP_TRANSFER_ENCODING "";

This will force-remove the variable in question, see uwsgi_param.

comment:2 by dener.kup@…, 2 years ago

Maxim, thanks for the reply! To be honest, I wasn't completely sure whether it is the nginx problem or uwsgi or werkzeug, as I can't find any good specification on uwsgi protocol.

I will go with this problem to werkzeug team and will post a link to the new issue on their side here.

We have applied similiar to yours workaround in werkzeug middleware. We don't remove all contents of Transfer-Encoding, but remove only 'chunked' words.

comment:3 by dener.kup@…, 2 years ago

Here is the link to the issue in werkzeug: https://github.com/pallets/werkzeug/issues/2346

Note: See TracTickets for help on using tickets.