#1009 closed defect (worksforme)
Upstream sent invalid header while reading response header from upstream
| Reported by: | Owned by: | ||
|---|---|---|---|
| Priority: | major | Milestone: | |
| Component: | nginx-core | Version: | 1.10.x |
| Keywords: | Upstream sent invalid header | Cc: | |
| uname -a: | Linux app.server 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | ||
| nginx -V: |
nginx version: nginx/1.10.0
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) built with OpenSSL 1.0.1e-fips 11 Feb 2013 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_perl_module=dynamic --add-dynamic-module=njs-1c50334fbea6/nginx --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --add-module=/home/builder/rpmbuild/BUILD/nginx-1.10.0/nginx-x-rid-header --with-http_v2_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' --with-ld-opt=-Wl,-E |
||
Description
Some background: in our solution we've got several Java microservices (Spring/Dropwizard + Jetty) behind Nginx which is used as a reverse proxy here.
Our microservices are stateless, we first log in with user/pass to our authorization service, receive a token and use that token to communicate with other microservices.
The problem occurs when we log in to authorization service and the response (with a token) is sent back to a customer - it stops on Nginx.
It has something in common with the size of a token (problem occurs when many scopes are added, with only a few it works), but increasing buffers (even a lot) didn't help.
We did a lot of checks - the communication works if we skip Nginx and call a Java service directly.
We also replaced Nginx with httpd just for a check and it also worked, however we prefer Nginx because of other features and performance reasons.
A full log line we see in Nginx logs:
2016/05/23 14:17:17 [error] 14503#0: *1 upstream sent invalid header while reading response header from upstream, client: 1.2.3.4, server: domain.com, request: "POST /uaa/login HTTP/1.1", upstream: "http://127.0.0.1:9999/uaa/login", host: "a.b.domain.com"
I'm attaching several files:
- our Nginx configuration
- logs from curls received after calling authorization service directly and with Nginx in between
We are using AWS EC2 service with Centos 7 box xlarge of size.
I have checked a lot of configuration changes, but no luck.
This is a big problem for us, I hope somebody can point me to solution.
I'm happy to provide additional info upon a request.
Attachments (3)
Change History (6)
by , 8 years ago
| Attachment: | curl_nginx.txt added |
|---|
follow-up: 3 comment:1 by , 8 years ago
The error suggests there is NUL character or stray CR character (not followed by LF) somewhere in headers returned. Try tcpdump'ing traffic between nginx and the backend to see what goes on on the wire.
comment:2 by , 8 years ago
| Resolution: | → worksforme |
|---|---|
| Status: | new → closed |
Unless you have some additional information to support the idea there is a bug in nginx, please use mailing list, as this doesn't looks relevant to Trac.
comment:3 by , 8 years ago
Replying to mdounin:
The error suggests there is NUL character or stray CR character (not followed by LF) somewhere in headers returned. Try tcpdump'ing traffic between nginx and the backend to see what goes on on the wire.
Thank you very much mdounin for this tip, as we knew what to look for. We used tcpdump and Wireshark to capture the traffic and found out that all CR LF were fine, however there was an error in our Java backend responsible for chunking headers. We set maximum size of a header to 4KB, but if it was exceeded, second header was filled up with NULLs up to 4KB. After fixing this case errors disappeard and everything works.
Curl with nginx as a reverse proxy