Opened 8 years ago

Closed 8 years ago

Last modified 7 weeks ago

#1143 closed defect (fixed)

http2 and auth_request corrupts first 32 bytes of POST request bodies longer than 8192 bytes

Reported by: jason.codeassassin.com@… Owned by:
Priority: major Milestone:
Component: nginx-core Version: 1.11.x
Keywords: Cc:
uname -a: Linux a4d477a7e472 3.13.0-85-generic #129-Ubuntu SMP Thu Mar 17 20:50:15 UTC 2016 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.11.6
built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.2)
built with OpenSSL 1.0.2g-fips 1 Mar 2016 (running with OpenSSL 1.0.2g 1 Mar 2016)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed'

Description

nginx 1.10.2 and 1.11.6 on Ubuntu 16.04 and Debian Stretch will corrupt the
first 32 bytes of a POST request body exceeding 8192 bytes when proxying from
a HTTP/2 client and also using an auth_request.

Using HTTP/1.1 instead of HTTP/2 stops the corruption.

Removing the auth_request configuration also stops the corruption.

POST request bodies of exactly 8192 bytes or less are not corrupted.

HTTP/2 clients tested include nghttp2 1.16, Chrome 55 for Windows, and
Firefox 51.0 for Windows. The client-side HTTP/2 over TLS traffic has been
captured, decrypted, and verified to contain the uncorrupted POST request body.

This was discovered affecting users in our Production environment and then reproduced in isolation with a minimal nginx configuration which has been published at https://github.com/jstangroome/nginx-http2-auth_request-post-corruption

The nginx.conf from the repository above is attached for convenience. Use nghttp https://127.0.0.1/with_auth_request --data=./a-file-larger-than-8192-bytes.txt or similar.

Attachments (1)

nginx.conf (2.3 KB ) - added by jason.codeassassin.com@… 8 years ago.

Download all attachments as: .zip

Change History (9)

by jason.codeassassin.com@…, 8 years ago

Attachment: nginx.conf added

comment:1 by jason.codeassassin.com@…, 8 years ago

May be related to #1128

comment:2 by Valentin V. Bartenev, 8 years ago

Thank you for the report.

Please, try the following patch:

diff -r 2c7a2d75938a -r a5ed250b880e src/http/v2/ngx_http_v2.c
--- a/src/http/v2/ngx_http_v2.c Mon Nov 21 16:49:19 2016 +0300
+++ b/src/http/v2/ngx_http_v2.c Thu Nov 24 14:28:07 2016 +0300
@@ -3575,6 +3575,9 @@ ngx_http_v2_read_request_body(ngx_http_r
         rb->buf = ngx_create_temp_buf(r->pool, (size_t) len);
 
     } else {
+        /* enforce writing body to file */
+        r->request_body_in_file_only = 1;
+
         rb->buf = ngx_calloc_buf(r->pool);
 
         if (rb->buf != NULL) {

Setting client_body_buffer_size 64k; or client_body_in_file_only on; should also help to workaround the problem.

comment:3 by jason.codeassassin.com@…, 8 years ago

Setting client_body_buffer_size 64k; definitely helps to workaround the issue. I'll try the patch soon.

comment:4 by jason.codeassassin.com@…, 8 years ago

Setting client_body_in_file_only on; helps workaround the problem.

Rebuilding nginx with the above patch and omitting the client_body_* directives also resolves the issue.

I think I'll use the client_body_in_file_only on; solution until the issue is patched in an official nginx build.

Version 0, edited 8 years ago by jason.codeassassin.com@… (next)

comment:5 by Valentin Bartenev <vbart@…>, 8 years ago

In 6805:52bd8cc17f34/nginx:

HTTP/2: fixed saving preread buffer to temp file (ticket #1143).

Previously, a request body bigger than "client_body_buffer_size" wasn't written
into a temporary file if it has been pre-read entirely. The preread buffer
is freed after processing, thus subsequent use of it might result in sending
corrupted body or cause a segfault.

comment:6 by Valentin V. Bartenev, 8 years ago

Resolution: fixed
Status: newclosed

comment:7 by Andrey Zelenkov <zelenkov@…>, 8 years ago

In 1088:83b7b3f8b6c5/nginx-tests:

Tests: added HTTP/2 tests with auth_request (ticket #1143).

Previously, a request body bigger than "client_body_buffer_size"
might result in sending corrupted body or cause a segfault.

comment:8 by Valentin Bartenev <vbart@…>, 8 years ago

In 6890:16487f9e6665/nginx:

HTTP/2: fixed saving preread buffer to temp file (ticket #1143).

Previously, a request body bigger than "client_body_buffer_size" wasn't written
into a temporary file if it has been pre-read entirely. The preread buffer
is freed after processing, thus subsequent use of it might result in sending
corrupted body or cause a segfault.

Note: See TracTickets for help on using tickets.