Opened 3 years ago
Last modified 3 years ago
#2341 closed defect
Sporadic "502 Bad Gateway" with reverse proxy — at Initial Version
| Reported by: | Owned by: | ||
|---|---|---|---|
| Priority: | major | Milestone: | |
| Component: | nginx-core | Version: | 1.16.x |
| Keywords: | reverse-proxy 502 | Cc: | |
| uname -a: | Linux itsrv2493.esrv.local 4.18.0-348.12.2.el8_5.x86_64 #1 SMP Mon Jan 17 07:06:06 EST 2022 x86_64 x86_64 x86_64 GNU/Linux | ||
| nginx -V: |
nginx version: nginx/1.16.1
built by gcc 8.2.1 20180905 (Red Hat 8.2.1-3) (GCC) built with OpenSSL 1.1.1 FIPS 11 Sep 2018 (running with OpenSSL 1.1.1k FIPS 25 Mar 2021) TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie |
||
Description
For unknown reasons I've got 502 responses. AFAIK there was no change anywhere, but this is not 100% certain. Though, anything must have changed, but nobody knows what this could be. At least it has worked for long time, but suddenly not anymore.
So, the problem is this: If I try to fetch 100 images with a http2 connection, many requests get a 502 response. If I do it with 20 images, all seems to be fine. The problem occur only if the reverse proxy is used. If the images gets fetched directly, there is never a 502 response. I tested that in browsers but also with curl on the command line.
The specified nginx version is 1.16.1 and the system is RHEL 8. But I tried it also with Fedora an nginx 1.20.2 in a VM with the same results.
Using curl on linux (mac is not able), you can try using the reverse proxy with this (you should see some 502 responses):
curl --http2 -Z -svo /dev/null https://demos.colop.com/creator4/media-cache/media/nio/{62052c6ce8288600019bdeef,620511a6e8288600019bdecf,6205036be8288600019bdebb,6204cda3e8288600016305cd,62043bd5e8288600016305c5,6203f219e8288600016305b6,6202c041e8288600016305a0,6201b0fbe828860001630593,6201a9abe82886000163058c,620190a8e82886000163057f,620182a3e828860001630578,620155d0e82886000163056b,62013eede828860001630561,620123b3e828860001630557,620110a2e82886000163054c,6200e643e828860001630544,6200347ce828860001630536,61fff0aae828860001630516,61fff044e82886000163050e,61ffef65e828860001630506,61ffc945e8288600016304f3,61ffafb7e8288600016304e8,61ffae28e8288600016304e3,61ff96bce8288600016304d5,61ff961de8288600016304cf,61fefe9ce8288600016304c3,61fef568e8288600016304b4,61fef0d5e8288600016304a8,61fed732e828860001630497,61feb647e828860001630485,61fd8ea6e828860001630467,61fd80fee82886000163044f,61fd7be2e828860001630449,61fd67a8e828860001630435,61fd44c7e828860001630426,61fd1107e828860001630416,61fc3e1ee828860001630407,61fc2a5ae8288600016303ff,61fc27c9e8288600016303f5,61fc2479e8288600016303ef,61fc1f94e8288600016303e7,61fc168ae8288600016303df,61fbdab4e8288600016303d4,61fad9aee8288600016303bf,61fa6510e8288600016303b0,61fa4f03e8288600016303a6,61fa43a6e828860001630398,61f9a2eee828860001630386,61f96d87e828860001630375,61f7fe3ce828860001630352,61f7fc56e82886000163034d,61f784cee828860001630339,61f708dae82886000163032b,61f70726e828860001630324,61f7044fe828860001630316,61f6eb8ae82886000163030b,61f6e7b9e8288600016302ff,61f6e25de8288600016302f1,61f6abb8e8288600016302c2,61f695e4e8288600016302a6,61f68b71e82886000163029d,61f5c1c0e82886000163026e,61f5bfefe828860001630267,61f5bec8e828860001630260,61f5a84ee82886000163025a,61f58de0e828860001630252,61f589c5e82886000163024a,61f53270e82886000163022c,61f4517de82886000163020c,61f44163e8288600016301fe,61f43e8de8288600016301f7,61f43c39e8288600016301f1,61f42bc6e8288600016301e8,61f429ade8288600016301e1,61f427b9e8288600016301d9,61f424f7e8288600016301d3,61f419bae8288600016301cb,61f3ff05e8288600016301c2,61f3fdeae8288600016301bc,61f3eb80e82886000163019c,61f3e295e828860001630196,61f3d79ce82886000163018e,61f3d00be828860001630185,61f3cf95e828860001630180,61f2ea44e82886000163015e,61f2dcebe828860001630157,61f2d5ace828860001630150,61f2c5b3e828860001630146,61f2bf9fe828860001630136,61f2a627e82886000163012e,61f2782ee828860001630124,61f1dff5e828860001630111,61f1c86fe828860001630104,61f1bbcae8288600016300e6,61f1b46ae8288600016300c1,61f1b094e8288600016300a6,61f18ec6e828860001630094,61f18c1ee82886000163008e,61f185f1e828860001630086,61f16f85e82886000163006b}
You can try the direct requests with the curl command (there shouldn't be any 502 responses):
curl --http2 -Z -svo /dev/null https://api.colop-online.com/service/creator/media-cache/media/nio/{62052c6ce8288600019bdeef,620511a6e8288600019bdecf,6205036be8288600019bdebb,6204cda3e8288600016305cd,62043bd5e8288600016305c5,6203f219e8288600016305b6,6202c041e8288600016305a0,6201b0fbe828860001630593,6201a9abe82886000163058c,620190a8e82886000163057f,620182a3e828860001630578,620155d0e82886000163056b,62013eede828860001630561,620123b3e828860001630557,620110a2e82886000163054c,6200e643e828860001630544,6200347ce828860001630536,61fff0aae828860001630516,61fff044e82886000163050e,61ffef65e828860001630506,61ffc945e8288600016304f3,61ffafb7e8288600016304e8,61ffae28e8288600016304e3,61ff96bce8288600016304d5,61ff961de8288600016304cf,61fefe9ce8288600016304c3,61fef568e8288600016304b4,61fef0d5e8288600016304a8,61fed732e828860001630497,61feb647e828860001630485,61fd8ea6e828860001630467,61fd80fee82886000163044f,61fd7be2e828860001630449,61fd67a8e828860001630435,61fd44c7e828860001630426,61fd1107e828860001630416,61fc3e1ee828860001630407,61fc2a5ae8288600016303ff,61fc27c9e8288600016303f5,61fc2479e8288600016303ef,61fc1f94e8288600016303e7,61fc168ae8288600016303df,61fbdab4e8288600016303d4,61fad9aee8288600016303bf,61fa6510e8288600016303b0,61fa4f03e8288600016303a6,61fa43a6e828860001630398,61f9a2eee828860001630386,61f96d87e828860001630375,61f7fe3ce828860001630352,61f7fc56e82886000163034d,61f784cee828860001630339,61f708dae82886000163032b,61f70726e828860001630324,61f7044fe828860001630316,61f6eb8ae82886000163030b,61f6e7b9e8288600016302ff,61f6e25de8288600016302f1,61f6abb8e8288600016302c2,61f695e4e8288600016302a6,61f68b71e82886000163029d,61f5c1c0e82886000163026e,61f5bfefe828860001630267,61f5bec8e828860001630260,61f5a84ee82886000163025a,61f58de0e828860001630252,61f589c5e82886000163024a,61f53270e82886000163022c,61f4517de82886000163020c,61f44163e8288600016301fe,61f43e8de8288600016301f7,61f43c39e8288600016301f1,61f42bc6e8288600016301e8,61f429ade8288600016301e1,61f427b9e8288600016301d9,61f424f7e8288600016301d3,61f419bae8288600016301cb,61f3ff05e8288600016301c2,61f3fdeae8288600016301bc,61f3eb80e82886000163019c,61f3e295e828860001630196,61f3d79ce82886000163018e,61f3d00be828860001630185,61f3cf95e828860001630180,61f2ea44e82886000163015e,61f2dcebe828860001630157,61f2d5ace828860001630150,61f2c5b3e828860001630146,61f2bf9fe828860001630136,61f2a627e82886000163012e,61f2782ee828860001630124,61f1dff5e828860001630111,61f1c86fe828860001630104,61f1bbcae8288600016300e6,61f1b46ae8288600016300c1,61f1b094e8288600016300a6,61f18ec6e828860001630094,61f18c1ee82886000163008e,61f185f1e828860001630086,61f16f85e82886000163006b}
I've tried to setup a reverse proxy with apache. I'm not sure if I have configured it correctly. But it seemed that there was no problem with apache.
My current (dumb) workaround it to let the browser load only 20 images or so at the same time.
What could be the problem? What could be the reason that things have worked and then suddenly not?
Note:
See TracTickets
for help on using tickets.
debug log