Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#1009 closed defect (worksforme)

Upstream sent invalid header while reading response header from upstream

Reported by: artursenk@… Owned by:
Priority: major Milestone:
Component: nginx-core Version: 1.10.x
Keywords: Upstream sent invalid header Cc:
uname -a: Linux app.server 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.10.0
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_perl_module=dynamic --add-dynamic-module=njs-1c50334fbea6/nginx --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --add-module=/home/builder/rpmbuild/BUILD/nginx-1.10.0/nginx-x-rid-header --with-http_v2_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' --with-ld-opt=-Wl,-E

Description

Some background: in our solution we've got several Java microservices (Spring/Dropwizard + Jetty) behind Nginx which is used as a reverse proxy here.
Our microservices are stateless, we first log in with user/pass to our authorization service, receive a token and use that token to communicate with other microservices.

The problem occurs when we log in to authorization service and the response (with a token) is sent back to a customer - it stops on Nginx.
It has something in common with the size of a token (problem occurs when many scopes are added, with only a few it works), but increasing buffers (even a lot) didn't help.

We did a lot of checks - the communication works if we skip Nginx and call a Java service directly.
We also replaced Nginx with httpd just for a check and it also worked, however we prefer Nginx because of other features and performance reasons.

A full log line we see in Nginx logs:
2016/05/23 14:17:17 [error] 14503#0: *1 upstream sent invalid header while reading response header from upstream, client: 1.2.3.4, server: domain.com, request: "POST /uaa/login HTTP/1.1", upstream: "http://127.0.0.1:9999/uaa/login", host: "a.b.domain.com"

I'm attaching several files:

  • our Nginx configuration
  • logs from curls received after calling authorization service directly and with Nginx in between

We are using AWS EC2 service with Centos 7 box xlarge of size.

I have checked a lot of configuration changes, but no luck.
This is a big problem for us, I hope somebody can point me to solution.
I'm happy to provide additional info upon a request.

Attachments (3)

curl_nginx.txt (967 bytes ) - added by artursenk@… 4 years ago.
Curl with nginx as a reverse proxy
curl_java.txt (7.3 KB ) - added by artursenk@… 4 years ago.
A direct curl to Java microservice
nginx.configuration.txt (3.8 KB ) - added by artursenk@… 4 years ago.
Nginx configuration

Download all attachments as: .zip

Change History (6)

by artursenk@…, 4 years ago

Attachment: curl_nginx.txt added

Curl with nginx as a reverse proxy

by artursenk@…, 4 years ago

Attachment: curl_java.txt added

A direct curl to Java microservice

by artursenk@…, 4 years ago

Attachment: nginx.configuration.txt added

Nginx configuration

comment:1 by Maxim Dounin, 4 years ago

The error suggests there is NUL character or stray CR character (not followed by LF) somewhere in headers returned. Try tcpdump'ing traffic between nginx and the backend to see what goes on on the wire.

comment:2 by Maxim Dounin, 4 years ago

Resolution: worksforme
Status: newclosed

Unless you have some additional information to support the idea there is a bug in nginx, please use mailing list, as this doesn't looks relevant to Trac.

in reply to:  1 comment:3 by artursenk@…, 4 years ago

Replying to mdounin:

The error suggests there is NUL character or stray CR character (not followed by LF) somewhere in headers returned. Try tcpdump'ing traffic between nginx and the backend to see what goes on on the wire.

Thank you very much mdounin for this tip, as we knew what to look for. We used tcpdump and Wireshark to capture the traffic and found out that all CR LF were fine, however there was an error in our Java backend responsible for chunking headers. We set maximum size of a header to 4KB, but if it was exceeded, second header was filled up with NULLs up to 4KB. After fixing this case errors disappeard and everything works.

Note: See TracTickets for help on using tickets.