Opened 4 years ago

Closed 4 years ago

Last modified 3 years ago

#1128 closed defect (worksforme)

HTTP/2 with ngx_http_auth_request_module causes some JSON payloads to become malformed.

Reported by: kcannon.gaikai.com@… Owned by:
Priority: major Milestone:
Component: other Version: 1.10.x
Keywords: Cc:
uname -a: Linux REDACTED 4.7.10-hardened-r2-base-7 #1 SMP Fri Nov 4 19:53:58 UTC 2016 x86_64 Intel Xeon E312xx (Sandy Bridge) GenuineIntel GNU/Linux
nginx -V: nginx version: nginx/1.10.2
built with OpenSSL 1.0.2j 26 Sep 2016
TLS SNI support enabled
configure arguments: --prefix=/usr --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error_log --pid-path=/run/nginx.pid --lock-path=/run/lock/nginx.lock --with-cc-opt=-I/usr/include --with-ld-opt=-L/usr/lib64 --http-log-path=/var/log/nginx/access_log --http-client-body-temp-path=/var/lib/nginx/tmp/client --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --with-http_v2_module --with-ipv6 --with-pcre --with-threads --with-http_auth_request_module --with-http_realip_module --add-module=external_module/headers-more-nginx-module-0.31 --add-module=external_module/ngx_devel_kit-0.3.0 --add-module=external_module/lua-nginx-module-0.10.6 --with-http_ssl_module --without-stream_upstream_hash_module --without-stream_upstream_least_conn_module --without-stream_upstream_zone_module --without-stream_upstream_hash_module --without-stream_upstream_least_conn_module --without-stream_upstream_zone_module --without-stream_upstream_hash_module --without-stream_upstream_least_conn_module --without-stream_upstream_zone_module --without-mail_imap_module --without-mail_pop3_module --without-mail_smtp_module --user=nginx --group=nginx

Description

We have an internal tool that requires all requests to be ran through auth_request. When we upgraded from 1.10.1 to 1.10.2, we noticed an increase in 400 bad request. When we would look at our nodeJS backend, we saw that the JSON body was corrupt. The request size does not matter and happens intermittently. We were able to fix the issue by rolling back to 1.10.1.

Example request body with malformed JSON in the beginning of the request:

14/Nov/2016:23:10:28 +0000 - POST /v2/heartbeat HTTP/2.0 --- h\x83}o\xBBm\x00\x00h\x83}o\xBBm\x00\x00n[1\xA0\x08\x00\x00n[1\xA0\x08\x00\x00st\x22:{\x22url\x22:\x22

14/Nov/2016:23:15:12 +0000 - POST /v2/heartbeat HTTP/2.0 --- {\x22events\x22:[{\x22code\x22:\x22727.1998\x22,\x22data\x22:

Attachments (4)

nginx-error-log.txt (125.0 KB ) - added by kcannon.gaikai.com@… 4 years ago.
nginx error log
app-error-log1.txt (143.0 KB ) - added by kcannon.gaikai.com@… 4 years ago.
app-error-log2.txt (192.8 KB ) - added by kcannon.gaikai.com@… 4 years ago.
app-error-log3.txt (86.2 KB ) - added by kcannon.gaikai.com@… 4 years ago.

Download all attachments as: .zip

Change History (9)

comment:1 by Valentin V. Bartenev, 4 years ago

Could you provide a debugging log?

by kcannon.gaikai.com@…, 4 years ago

Attachment: nginx-error-log.txt added

nginx error log

by kcannon.gaikai.com@…, 4 years ago

Attachment: app-error-log1.txt added

by kcannon.gaikai.com@…, 4 years ago

Attachment: app-error-log2.txt added

by kcannon.gaikai.com@…, 4 years ago

Attachment: app-error-log3.txt added

comment:2 by Valentin V. Bartenev, 4 years ago

Are you able to reproduce the issue without 3-rd party modules?

comment:3 by kcannon.gaikai.com@…, 4 years ago

We are not able to reproduce this issue without using 3-rd party modules. We have seen the same issue with access_by_lua as well. Is it typical for a patch release to break 3-rd party modules?

in reply to:  3 comment:4 by Valentin V. Bartenev, 4 years ago

Resolution: worksforme
Status: newclosed

I wasn't able to reproduce it too with HTTP/2 and the auth request module.

Replying to kcannon.gaikai.com@…:

Is it typical for a patch release to break 3-rd party modules?

Unfortunately some 3rd-party modules abuse nginx internal API interfaces. They tries to access and modify structures that they are not supposed to. As a result, such modules may get broken even due to some unrelated bug fixes.

comment:5 by kcannon.gaikai.com@…, 3 years ago

Looks like this was fixed in 1.10.3 (https://nginx.org/en/CHANGES-1.10)

Bugfix: when using HTTP/2 and the "limit_req" or "auth_request"

directives client request body might be corrupted; the bug had
appeared in 1.10.2.

Note: See TracTickets for help on using tickets.