Opened 4 years ago

Last modified 4 years ago

#1950 closed defect

nginx/1.17.9 randomly truncating some large proxy responses — at Version 1

Reported by: Artem Russakovskii Owned by:
Priority: major Milestone:
Component: nginx-core Version:
Keywords: truncate, truncation, cut off, response Cc:
uname -a: Linux 5.4.10-x86_64-linode132 #1 SMP PREEMPT Thu Jan 9 21:17:12 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx/1.17.9
built by gcc 7.4.0 (SUSE Linux)
built with OpenSSL 1.1.0i-fips 14 Aug 2018 (running with OpenSSL 1.1.1f 31 Mar 2020)
TLS SNI support enabled
configure arguments: --prefix=/usr/ --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/run/ --lock-path=/run/nginx.lock --http-client-body-temp-path=/var/lib/nginx/tmp/ --http-proxy-temp-path=/var/lib/nginx/proxy/ --http-fastcgi-temp-path=/var/lib/nginx/fastcgi/ --http-uwsgi-temp-path=/var/lib/nginx/uwsgi/ --http-scgi-temp-path=/var/lib/nginx/scgi/ --user=nginx --group=nginx --without-select_module --without-poll_module --with-threads --with-file-aio --with-ipv6 --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_auth_request_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_slice_module --with-http_stub_status_module --with-http_perl_module=dynamic --with-perl=/usr/bin/perl --with-mail=dynamic --with-mail_ssl_module --with-stream=dynamic --with-stream_ssl_module --with-stream_realip_module --with-stream_ssl_preread_module --with-pcre --with-pcre-jit --with-libatomic --with-compat --add-module=nginx_upstream_check_module-0.3.0 --add-dynamic-module=ngx-fancyindex-0.4.2 --add-dynamic-module=headers-more-nginx-module-0.33 --add-dynamic-module=nginx-rtmp-module-1.2.1 --add-dynamic-module=ngx_http_geoip2_module-3.3 --with-cc-opt='-fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g -fPIC -D_GNU_SOURCE' --with-ld-opt='-Wl,-z,relro,-z,now -pie'

Description (last modified by Artem Russakovskii)


We're using nginx/1.17.9 on 5 servers, and one of them started behaving oddly about a week ago, randomly truncating some proxy responses - at least the ones we noticed are specifically WordPress xmlrpc responses from Apache running on the same host, which nginx proxies to.

A few things stand out about this.

  1. We haven't noticed any other data getting truncated similarly, only these xmlrpc XML API responses (at least not yet).
  1. After we narrowed it down to nginx (querying Apache directly produced no truncation), I decided to first restart nginx. After the restart, the issue went away (at least for now). None of the other 4 web servers running nginx and apache in the same manner have ever exhibited this behavior.
  1. The machine has plenty of space (1.3TB), so it's not an issue of lack of space.
  1. Here's the most interesting bit. This machine was rebooted about a week ago, which is exactly when the issue started.

The 2nd interesting bit is the reboot kicked in a new kernel: 5.4.10-x86_64-linode132.

All the other machines haven't been rebooted yet and are running 5.1.17-x86_64-linode128.

  1. The response is about 600KB. Sometimes ~500KB was returned, sometimes less (~400KB). Sometimes the full response was there.

Is this a known issue? Do you have any ideas about why it may have happened?

Change History (1)

comment:1 by Artem Russakovskii, 4 years ago

Description: modified (diff)
Note: See TracTickets for help on using tickets.