Opened 4 years ago
Closed 3 years ago
#2111 closed defect (worksforme)
Worker dumps core with image_filter test, proxy and HTML response from upstream
| Reported by: | Owned by: | ||
|---|---|---|---|
| Priority: | major | Milestone: | |
| Component: | nginx-module | Version: | 1.19.x |
| Keywords: | image_filter | Cc: | |
| uname -a: | Linux xxxx 4.18.0-240.1.1.el8_3.x86_64 #1 SMP Thu Nov 19 17:20:08 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | ||
| nginx -V: |
nginx version: nginx/1.18.0
built by gcc 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC) built with OpenSSL 1.1.1c FIPS 28 May 2019 (running with OpenSSL 1.1.1g FIPS 21 Apr 2020) TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie' |
||
Description
nginx worker process dumps core (and drops connection) if the following prerequisites are satisfied:
- Use nginx as reverse proxy for URIs with Regex location
- Enable caching
- Configure a rewrite rule inside that location
- Add "image_filter test" to location
- Configure a PNG file as error page for error 415, i.e.
- Request (GET) an URI through nginx, which leads to a 404 error on upstream, Upstream will send HTML content with correct content type "text/html"
-- Requests shall be made with curl or wget - for unknown reasons that bug will not trigger when you use a browser like Firefox. Also, this will not work with "HEAD" requests!
- Worker will core dump
I have traced this down to the use of "image_filter test".
This happens with HTTP/2 as well as with HTTP/1.1.
If I enable "proxy_intercept" to deliver my custom error page, the problem does not occur.
Example config:
error_page 415 404 403 500 502 503 504 /errorpages/image-not-found.png
location ~* /images/(.*).jpg$ {
rewrite /images/(?<imgid>(.*).jpg$ /media/get/$imgid break;
image_filter test;
proxy_pass .....
}
Request:
curl -v 'https://domain/images/404.jpg' > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
39 83829 39 32768 0 0 11175 0 0:00:07 0:00:02 0:00:05 11172
curl: (18) transfer closed with 51061 bytes remaining to read
Upstream delivers a HTML response instead of an image:
HTTP/1.1 404 Not Found
Date: Fri, 18 Dec 2020 15:33:33 GMT
Server: Apache
Cache-Control: no-cache, private
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
327
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8" />
<meta name="robots" content="noindex,nofollow,noarchive" />
<title>An Error Occurred: Not Found</title>
.......
Log says:
[alert] 50253#50253: worker process 50260 exited on signal 11 (core dumped)
Stack trace:
#0 0x0000557821885ccc ngx_http_file_cache_update (nginx) #1 0x000055782187c2df ngx_http_upstream_process_request (nginx) #2 0x000055782187b662 ngx_http_upstream_handler (nginx) #3 0x00005578218543ae ngx_epoll_process_events (nginx) #4 0x000055782184a7da ngx_process_events_and_timers (nginx) #5 0x0000557821852639 ngx_worker_process_cycle (nginx) #6 0x0000557821850b2f ngx_spawn_process (nginx) #7 0x0000557821851cf0 ngx_start_worker_processes (nginx) #8 0x0000557821853007 ngx_master_process_cycle (nginx) #9 0x0000557821829a56 main (nginx) #10 0x00007f54d2d347b3 __libc_start_main (libc.so.6) #11 0x0000557821829abe _start (nginx)
Change History (2)
comment:1 by , 4 years ago
comment:2 by , 3 years ago
| Resolution: | → worksforme |
|---|---|
| Status: | new → closed |
Feedback timeout. Unfortunately, it is not possible to reproduce the issue from the information provided.
Note:
See TracTickets
for help on using tickets.
Could you please provide details on how
/errorpages/image-not-found.pngis handled? A debug log might be helpful, too.Overall this looks like a variant of well known filter finalization issues. Some of the issues were addressed in 07f028df3879 and 5abf5af257a7, but certainly there may be others, especially if caching is used. (At some point I've tried to submit a rather radical patch to address filter finalization issues with cache, but the patch was rejected by Igor since it contradicts his idea on how cache is expected to be used.)