Opened 4 years ago

Closed 4 years ago

Last modified 2 years ago

#1064 closed defect (invalid)

NGINX DNS CACHING problem.

Reported by: CorbanR@… Owned by:
Priority: minor Milestone:
Component: nginx-core Version: 1.11.x
Keywords: Cc:
uname -a: Linux 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.11.3
built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3)
built with OpenSSL 1.0.1f 6 Jan 2014
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_perl_module=dynamic --add-dynamic-module=debian/extra/njs-0.1.0/nginx --with-threads --with-stream --with-stream_ssl_module --with-stream_geoip_module=dynamic --with-http_slice_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --with-http_v2_module --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--as-needed'

Description

With the following Configuration

`stream {

resolver 169.254.169.253 ipv6=off; #AWS DNS
error_log /var/log/nginx/nginx-elb.log info;
upstream up_prod_tcp_80 {

server someawselb.us-east-1.elb.amazonaws.com:80 weight=3 max_fails=10 fail_timeout=60s;
server someawselb2.us-east-1.elb.amazonaws.com:80 max_fails=10 fail_timeout=60s;

}

upstream up_prod_tcp_443 {

server someawselb.us-east-1.elb.amazonaws.com:443 weight=3 max_fails=10 fail_timeout=60s;
server someawselb2.us-east-1.elb.amazonaws.com:443 max_fails=10 fail_timeout=60s;

}

server {

listen 80;
proxy_pass up_prod_tcp_80;

}
server {

listen 443;
proxy_pass up_prod_tcp_443;

}

}`

I would expect NGINX to cache the DNS based on the TTL of the load balancer, (AWS ELB's are 60s). Seems like anytime the AWS ELB Public IP changes I get upstream timed out (110: Connection timed out) while connecting to upstream This is due to NGINX trying to connect to an OLD DNS entry.

Change History (2)

comment:1 by Maxim Dounin, 4 years ago

Resolution: invalid
Status: newclosed

This is expected behaviour. Names are resolved to IP addresses while loading a configuration. To re-resolve them on changes you have to instruct nginx to reload the configuration. Alternatively, you can:

  • use proxy_pass with variables and the resolver directive to force nginx to resolve names at run-time;
  • use the resolve parameter of the server directive in the upstream block to trigger periodic resolution of a name (available as part of the commercial subscription).

For further questions on nginx please consider using mailing list instead.

comment:2 by colinbjohnson@…, 2 years ago

There is an additional workaround for folks who do not wish to "utilize proxy_pass with variables and the ​resolver directive" and perform DNS resolution on a schedule (note that this workaround would cause outages) - the AWS Network Load Balancer supports static IP addressing and "internet-facing" and "internal" schemes. The Network Load Balancer solution eliminates the ability to utilize security groups for particular resources.

Note: See TracTickets for help on using tickets.