Opened 4 weeks ago

Closed 4 weeks ago

Last modified 4 weeks ago

#2072 closed defect (invalid)

memcached_pass with localhost:11211 sometimes ignore port and name resolve.

Reported by: partizanes@… Owned by:
Priority: minor Milestone:
Component: nginx-module Version: 1.16.x
Keywords: Cc:
uname -a: Linux * 2.6.32-754.30.2.el6.x86_64 #1 SMP Wed Jun 10 11:14:37 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.16.1
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-http_geoip_module --add-dynamic-module=/opt/testcookie-nginx-module/ --add-dynamic-module=/opt/nginx-http-rdns/ --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

Description

Problem with use

memcached_pass localhost:11211;


It work , but sometimes ignore port and name resolve. When it`s work in logs varilable upstream contains correct path.

upstream: "memcached://127.0.0.1:11211"

But if have error "no live upstreams while connecting to upstream" in upstream:

upstream: "memcached://localhost"

For debug i use several log file in problem location:

Work(upstream is correct):

2020/11/01 17:17:56 [info] 7904#7904: *73653192 key: "DesktopMenu/IndexController.php/ssi/DesktopMenu/IndexController.php?type=ajax-1" was not found by memcached while reading response header from upstream, client: *, server: *, request: "GET /ssi/DesktopMenu/IndexController.php?type=ajax HTTP/1.1", upstream: "memcached://127.0.0.1:11211", host: "*", referrer: "*"

Don`t work(upstream if wrong):

2020/11/01 17:20:55 [error] 7918#7918: *73679779 no live upstreams while connecting to upstream, client: *, server: *, request: "GET /ssi/DesktopMenu/IndexController.php?type=ajax HTTP/1.1", upstream: "memcached://localhost", host: "*", referrer: "*"

Memcached totat request rate is 15 per second. Using 127.0.0.1:11211 instead localhost:11211 is resolve the problem.

Memcached limits:
memcached -d -p 11211 -u memcached -m 2048 -c 4096 -P /var/run/memcached/memcached.pid -l 127.0.0.1 -a 0766 -n 16 -f 1.05

/etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

Change History (3)

comment:1 by Maxim Dounin, 4 weeks ago

Resolution: invalid
Status: newclosed

The no live upstreams error means that nginx wasn't able to select a server to connect to because all servers are marked unavailable due to previous errors, see max_fails. The error log line is expected to contain the upstream name rather than an IP address since all available IP addresses are marked unavailable.

Note that this logic is only used if the name maps to more than one address. That is, if it happens to you with localhost, it means that localhost resolves more than just 127.0.0.1 - most likely, it also resolves to ::1 as well.

To fix this, consider:

  • Configuring an upstream server group explicitly, and specify max_fails and fail_timeout you want to be applied.
  • Using a name which resolves into a single address, such as localhost4.

Also it might be good idea to find why errors happen when accessing the memcached server, so it is marked unavailable. Error log should contain details.

comment:2 by partizanes@…, 4 weeks ago

Memcached is minimally used and absolutely accessible.

I will assume that sometimes localhost resolves to an ipv6 address where memcached is not available.

/etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

When using a direct address, no problems with the availability of memcached were recorded even once.It seems to me this is not entirely obvious behavior.

comment:3 by Maxim Dounin, 4 weeks ago

Given that localhost is defined to be 127.0.0.1 and ::1 on your system, it always resolves to both IPv4 and IPv6 address. And nginx is smart enough to find out that IPv6 address is not responding and mostly use IPv4 one (trying non-working IPv6 address only once per 10 seconds, default fail_timeout). But the no live upstreams error in your logs suggests there are errors when using IPv4 address as well, hence the recommendation to find out why errors happen when accessing the memcached server. You should be able to see corresponding errors in the error log at the error level.

Note: See TracTickets for help on using tickets.