browser cannot correctly decode http response headers when http2 is used

[val-kulkov@…]

When http2 is enabled, Google Chrome and Firefox cannot correctly decode http2 response headers received from Nginx. The response headers appear scrambled. At the same time, both browsers decode the body of the response correctly.

Since the browser cannot correctly decode http2 response headers, the browser displays "ERR_SPDY_COMPRESSION_ERROR" error ("gzip on;" in nginx.conf) or the "ERR_HTTP2_PROTOCOL_ERROR" ("gzip off;").

This issue has been observed in OpenWrt by multiple users: ​https://github.com/openwrt/packages/issues/8988

A Wireshark packet capture of a conversation between Google Chrome and Nginx (with SSLKEYLOGFILE environment variable set to capture SSL keys) is attached. To decode and view the captured SSL stream in Wireshark, right-click on a TLSv1.3 packet in the packets pane, select "Protocol Preferences" -> "(Pre)-Master-Secret log filename...", click on "Browse..." button next to "(Pre)-Master-Secret log filename..." and select the attached "session_keys.log" file.

Packet 36 is the browser's http2 request to Nginx. Everything looks normal in it. Packet 39 contains the http2 response from Nginx, and that is where things are not normal. The first header in the response looks good: ":status: 200 OK", but then all remaining headers except content-type and content-length appear scrambled.

It is very odd that the body of http2 response from Nginx in packet 41 appears just fine: the browser is able to correctly decode the body of http2 response.

This problem manifests itself on the OpenWrt x86_64 platform, as noted in the OpenWrt issue mentioned above. Nginx on non-x86_64 OpenWrt platforms appear to work fine.

[val-kulkov@…]

Hello admins,

I cannot attach any file to this ticket. I get this error message when uploading a file:

Error: Bad Request
Missing or invalid form token. Do you have cookies enabled?
TracGuide — The Trac User and Administration Guide

Please can someone fix Trac and let me know so that I can attach two files to this ticket, one with Wireshark packet capture and one with SSL keys?

[val-kulkov@…]

As a faster alternative, I can email both files to someone who can manually update this ticket at Trac. In this case, I simply need an email address.

[Maxim Dounin]

Are you able to reproduce the problem without 3rd party patches? Configure arguments clearly show that you are using a patched nginx version, and 3rd party crossbuild patches as seen elsewhere were reported to cause incorrect builds more than once. Please try a clean version as available from ​http://nginx.org/en/download.html instead, and build it natively.

(I've reported issues with attachments, it should be fixed soon. But in this particular case there is no need to attach anything unless there is a confirmation that the problem exists without any 3rd party patches.)

[val-kulkov@…]

There are only six patches that are applied to Nginx while cross-compiling for OpenWrt. All of them appear fairly trivial:

​https://github.com/openwrt/packages/tree/master/net/nginx/patches

For example, 101-feature_test_fix.patch disables run tests which routinely fail in a cross-compilation environment.

I will check if all these patches are still necessary and report the results back here.

[val-kulkov@…]

Hi Maxim,

Patch 102-sizeof_test_fix.patch (see the link above) is required because Nginx's configure attempts to compile $NGX_AUTOTEST.c (line 32 in auto/types/sizeof) and then execute it which makes no sense in a cross-compilation environment. The host cannot execute the target's binary code without an emulator.

If you have any suggestion how I should determine the size of $ngx_type on the target platform without executing the target platform's binary code, I would much appreciate it.

[Maxim Dounin]

To make things clear: nginx does not support cross-compilation. If you are trying to cross-compile nginx, it is your responsibility to ensure configure results are correct. Given that cross-compilation results in a misbehaving binary, the first thing to test is a native compilation, without any patches. Are you able to reproduce the problem with a native build of vanilla nginx, without any patches?

[val-kulkov@…]

Hi Maxim,

It is not possible to compile Nginx for OpenWrt without these patches:

101-feature_test_fix.patch
102-sizeof_test_fix.patch
103-sys_nerr.patch

This is because the host platform is unable to execute the target platform binaries.

The way I see it, 101-feature_test_fix.patch and 103-sys_nerr.patch are harmless and IMO they could not possibly be the culprits. The 200 and 300 series patches do not modify the code. They only modify configuration files and therefore they are not suspects.

102-sizeof_test_fix.patch is a possible suspect, but it looks okay to me. It uses a clever hack to determine the size of int. Here is the relevant part of auto/types/sizeof after applying the patch:

cat << END > $NGX_AUTOTEST.c

#include <sys/types.h>
#include <sys/time.h>
$NGX_INCLUDE_UNISTD_H
#include <signal.h>
#include <stdio.h>
#include <sys/resource.h>
$NGX_INCLUDE_INTTYPES_H
$NGX_INCLUDE_AUTO_CONFIG_H

char object_code_block[] = {
	'\n', 'e', '4', 'V', 'A',
	'0', 'x', ('0' + sizeof($ngx_type)),
	'Y', '3', 'p', 'M', '\n'
};

int main(void) {
    printf("dummy use of object_code_block to avoid gc-section: %c", object_code_block[0]);
    return 0;
}

END


ngx_test="$CC $CC_TEST_FLAGS $CC_AUX_FLAGS \
          -o $NGX_AUTOTEST $NGX_AUTOTEST.c $NGX_LD_OPT $ngx_feature_libs"

eval "$ngx_test >> $NGX_AUTOCONF_ERR 2>&1"


if [ -x $NGX_AUTOTEST ]; then
    ngx_size=`sed -ne 's/^e4VA0x\(.\)Y3pM$/\1/p' < $NGX_AUTOTEST`
    echo " $ngx_size bytes"
fi

Note that it is the ('0' + sizeof($ngx_type)) construct in object_code_block[] that is used to determine the size of int. "sed" utility then looks for the sequence of characters defined in object_code_block and obtains value of ('0' + sizeof($ngx_type)). Do you see any issue with this approach?

[val-kulkov@…]

Hi Maxim,

It seems that I found the root cause of the current problem. The endianness of the target platform cannot be correctly determined in auto/endianness because the current test relies on the execution of the code on the target platform during cross-compilation. The "big endian" is the fallback option and if the target platform is "big endian" then everything works. The "little endian" platforms predictably fail.

I am working with the OpenWrt community to see if we can propose a method to reliably identify endianness of the target platform during cross-compilation.

[Maxim Dounin]

Certainly it is possible to compile nginx for OpenWrt without any patches. You have do it natively, using a native toolchain. On the target platform itself, without using cross-compilation.

Anyway, thanks for confirming that the problem is due to incorrect cross-compilation patches. Closing this.

[val-kulkov@…]

OpenWrt is a platform for embedded devices using minimal amounts of RAM and non-volatile storage space. On many of these devices there is simply not enough RAM or storage space to install and use a native toolchain. For this reason, OpenWrt targets are normally built by cross-compilation on host workstations that have enough RAM and disk space.

Anyway, thank you for your consideration.