Response headers in body

[adrienb4@…]

I dont understand why the response headers are in the body with 'proxy_path' option. I woul'd like to use this with OpenWRT 15.05

example :

---

Server: nginx/1.6.2
Date: Thu, 11 Feb 2016 14:02:17 GMT
Content-Type: text/plain
Content-Length: 6
Last-Modified: Fri, 05 Feb 2016 14:38:29 GMT
Connection: close
ETag: "56b4b3e5-6"
Accept-Ranges: bytes

prout

---

I have no problems with the last OpenWRT sources.

Any idea ?

Thank

Adrien

[adrienb4@…]

2016/02/11 15:47:24 [error] 1223#0: *13 upstream sent no valid HTTP/1.0 header while reading response header from upstream, client: 192.168.1.146, server: 192.168.1.1, request: "GET /test.txt HTTP/1.1", upstream: "​http://192.168.6.40:80/test.txt", host: "192.168.1.1"

[Maxim Dounin]

The message suggests that upstream server returned a response without a status line, hence nginx interpreted it as HTTP/0.9 and returned to client. How the upstream response looks like?

[adrienb4@…]

Sorry but I dont understand what do you mean by "How the upstream response looks like?"

[Valentin V. Bartenev]

Replying to adrienb4@…:

Sorry but I dont understand what do you mean by "How the upstream response looks like?"

The error message is clear upstream sent no valid HTTP/1.0 header, so the reasonable question is what the upstream actually sent in response to nginx?

[adrienb4@…]

The upstream works correctly with another nginx in proxy_pass. It's just when I compile it in OpenWRT.

[Valentin V. Bartenev]

Replying to adrienb4@…:

The upstream works correctly with another nginx in proxy_pass. It's just when I compile it in OpenWRT.

There's nothing wrong with the error message and the nginx behavior in this case. If you care to debug the issue, you should provide more information.

[adrienb4@…]

See attached file. It's a tcpdump file with the traffic between nginx (192.168.1.1) and the upstream (192.168.6.40)

[Valentin V. Bartenev]

So the response from your upstream server is not a valid HTTP/1.x response, it has a missing status line:

$ tcpflow -c -D -r ngin_tcpdump.pcap
192.168.001.146.37074-192.168.001.001.00080: 
0000: 4745 5420 2f74 6573 742e 7478 7420 4854 5450 2f31 2e31 0d0a 486f 7374 3a20 3139  GET /test.txt HTTP/1.1..Host: 19
0020: 322e 3136 382e 312e 310d 0a43 6f6e 6e65 6374 696f 6e3a 206b 6565 702d 616c 6976  2.168.1.1..Connection: keep-aliv
0040: 650d 0a43 6163 6865 2d43 6f6e 7472 6f6c 3a20 6d61 782d 6167 653d 300d 0a41 6363  e..Cache-Control: max-age=0..Acc
0060: 6570 743a 2074 6578 742f 6874 6d6c 2c61 7070 6c69 6361 7469 6f6e 2f78 6874 6d6c  ept: text/html,application/xhtml
0080: 2b78 6d6c 2c61 7070 6c69 6361 7469 6f6e 2f78 6d6c 3b71 3d30 2e39 2c69 6d61 6765  +xml,application/xml;q=0.9,image
00a0: 2f77 6562 702c 2a2f 2a3b 713d 302e 380d 0a55 7067 7261 6465 2d49 6e73 6563 7572  /webp,*/*;q=0.8..Upgrade-Insecur
00c0: 652d 5265 7175 6573 7473 3a20 310d 0a55 7365 722d 4167 656e 743a 204d 6f7a 696c  e-Requests: 1..User-Agent: Mozil
00e0: 6c61 2f35 2e30 2028 5831 313b 204c 696e 7578 2078 3836 5f36 3429 2041 7070 6c65  la/5.0 (X11; Linux x86_64) Apple
0100: 5765 624b 6974 2f35 3337 2e33 3620 284b 4854 4d4c 2c20 6c69 6b65 2047 6563 6b6f  WebKit/537.36 (KHTML, like Gecko
0120: 2920 4368 726f 6d65 2f34 372e 302e 3235 3236 2e31 3036 2053 6166 6172 692f 3533  ) Chrome/47.0.2526.106 Safari/53
0140: 372e 3336 0d0a 4163 6365 7074 2d45 6e63 6f64 696e 673a 2067 7a69 702c 2064 6566  7.36..Accept-Encoding: gzip, def
0160: 6c61 7465 2c20 7364 6368 0d0a 4163 6365 7074 2d4c 616e 6775 6167 653a 2066 722d  late, sdch..Accept-Language: fr-
0180: 4652 2c66 723b 713d 302e 382c 656e 2d55 533b 713d 302e 362c 656e 3b71 3d30 2e34  FR,fr;q=0.8,en-US;q=0.6,en;q=0.4
01a0: 0d0a 0d0a                                                                        ....

192.168.001.001.00080-192.168.001.146.37074: 
0000: 5365 7276 6572 3a20 6e67 696e 782f 312e 362e 320d 0a44 6174 653a 2054 6875 2c20  Server: nginx/1.6.2..Date: Thu, 
0020: 3131 2046 6562 2032 3031 3620 3134 3a30 323a 3137 2047 4d54 0d0a 436f 6e74 656e  11 Feb 2016 14:02:17 GMT..Conten
0040: 742d 5479 7065 3a20 7465 7874 2f70 6c61 696e 0d0a 436f 6e74 656e 742d 4c65 6e67  t-Type: text/plain..Content-Leng
0060: 7468 3a20 360d 0a4c 6173 742d 4d6f 6469 6669 6564 3a20 4672 692c 2030 3520 4665  th: 6..Last-Modified: Fri, 05 Fe
0080: 6220 3230 3136 2031 343a 3338 3a32 3920 474d 540d 0a43 6f6e 6e65 6374 696f 6e3a  b 2016 14:38:29 GMT..Connection:
00a0: 2063 6c6f 7365 0d0a 4554 6167 3a20 2235 3662 3462 3365 352d 3622 0d0a 4163 6365   close..ETag: "56b4b3e5-6"..Acce
00c0: 7074 2d52 616e 6765 733a 2062 7974 6573 0d0a 0d0a 7072 6f75 740a                 pt-Ranges: bytes....prout.

You should investigate what's going on with your upstream server, or if some MITM corrupts the response.

[Valentin V. Bartenev]

Replying to adrienb4@…:

See attached file. It's a tcpdump file with the traffic between nginx (192.168.1.1) and the upstream (192.168.6.40)

Well, the attached ngin_tcpdump.pcap actually contains traffic between 192.168.1.146 (acts as a client) and 192.168.1.1 (acts as a server).

[adrienb4@…]

Thank. And how I can add this status line ?

[Valentin V. Bartenev]

Are you sure that you have provided the right tcpdump? See my note about the IPs above.

[adrienb4@…]

Yes it's the right tcpdump file. Upstream server is on another interface (wan) of my routeur and tcpdump listen to lan interface

[Maxim Dounin]

As per the log message provided, your upstream server is at 192.168.6.40, and nginx is on 192.168.1.1. The dump provided contains traffic between client (192.168.1.146) and nginx (192.168.1.1), but not between nginx (192.168.1.1) and the upstream server (192.168.6.40). You'll have to look at a dump between nginx and the upstream server to find out if it's the upstream server problem or something wrong happens in nginx.

Note well that nginx version you are using is:

• outdated;
• contains at least several 3rd party modules;
• uses 3rd party patches for crossbuilding, which may cause problems;

If you'll find out that something wrong happens in nginx, I would recommend to try native build without 3rd party modules to see if it helps.

[jeroen.roovers.airfi.aero@…]

I may be seeing a variant of this problem.

I originally found this problem somewhere last year. I was using 1.8.1 at the time (compiled via Buildroot) and I only observed it when switching from GCC 4.8.5 to 4.9.3.

Doing a simple curl request on the backend server (I ruled that out since I can see it with the couchdb/futon interface as well as a nodejs based web service) returns "HTTP/1.1 200 OK" just fine, but going through nginx I see the same message logged as above.

nginx seems to miscompile, then.

Meanwhile I have upgraded (beyond what Buildroot provides) to 1.9.12 and I still see this issue when compiling with GCC 4.9.3.

[jeroen.roovers.airfi.aero@…]

Oh, I'm still on 1.9.11. I'll test with 1.9.12 shortly but I don't see significant changes in ngx_http_parse.c where the problem likely originates.

nginx version: nginx/1.9.11
built by gcc 4.9.3 (Buildroot 2016.02-00588-gb5b44d5-dirty)
built with OpenSSL 1.0.2g 1 Mar 2016
TLS SNI support enabled
configure arguments: --crossbuild=Linux::arm --with-cc=/home/jer/airfi/airfi-base/buildroot-2016.02-moon/output/host/usr/bin/arm-buildroot-linux-gnueabihf-gcc --with-cpp=/home/jer/airfi/airfi-base/buildroot-2016.02-moon/output/host/usr/bin/arm-buildroot-linux-gnueabihf-gcc --with-cc-opt='-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -fstack-protector-strong' --with-ld-opt='-Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu' --with-ipv6 --prefix=/usr --conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/sbin/nginx --pid-path=/var/run/nginx.pid --lock-path=/var/run/lock/nginx.lock --user=www-data --group=www-data --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/tmp/nginx/client-body --http-proxy-temp-path=/var/tmp/nginx/proxy --http-fastcgi-temp-path=/var/tmp/nginx/fastcgi --http-scgi-temp-path=/var/tmp/nginx/scgi --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi --with-file-aio --with-pcre --with-select_module --with-poll_module --with-http_v2_module --with-http_ssl_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_secure_link_module --with-http_realip_module --with-http_addition_module --with-http_mp4_module --with-http_auth_request_module --with-http_random_index_module --with-http_degradation_module --with-http_stub_status_module --without-http_uwsgi_module --without-http_memcached_module

[jeroen.roovers.airfi.aero@…]

checking for system byte ordering ... big endian

No, that's not big-endian.

[jeroen.roovers.airfi.aero@…]

Also, this test is run on the host, not the target, so cross-compiling is a problem anyway.

Still, I don't see why it should turn out a big-endian result on a little-endian host.

And indeed, with a forced -DNGX_HAVE_LITTLE_ENDIAN in CFLAGS I get the desired result.

[Maxim Dounin]

So this looks like a problem with 3rd party cross-compilation patches as previously suggested in comment:13, not with nginx itself. You may want to report the problem to Buildroot folks. Thanks for the detailed feedback, closing this as invalid.

[jeroen.roovers.airfi.aero@…]

I strongly disagree with your decision.

There are two problems here: one is a bad nginx configure test. The other is a miscompilation using GCC 4.9 which may or may not be a problem in GCC.

The nginx test can be fixed: compilers can be tested for the endianness of their target. What your test is doing is take the compiler, compile a test program and then try to run it. If running it fails (because the resulting program cannot be executed on the host) you assume a big endian system.

Either you don't support cross-compiling at all, or you fix the test. It's up to you, and as long as everyone knows you don't support cross-compiling, then that's fine too.

I wanted to open a new issue about compiling with GCC 4.9, but I'm not so sure that will help now.

[Maxim Dounin]

We don't support cross-compilation. An attempt to cross-compile nginx will fail with a more or less obvious message way before the byte ordering test. You are using 3rd party patches for cross-compilation support, and the problem you are facing is in 3rd party patches - they partially allow cross-compilation, but fail to properly address all cases where nginx tries to execute compiled code. As already suggested, please report this problem to the patch authors.

As for compiling with GCC 4.9, please see ticket #912. GCC 4.9 is known to miscompile for ARM at least when using -O2.