Nginx DNS cache issue. ngx_http_core_module valid config is not working.

[aswin020@…]

Setup:

Ec2 instance with Nginx 1.12.2 behind an AWS Load balancer. The load balancer has an Idle timeout of 60 seconds. Upstream is again another Loadbalancer. The DNS TTL of the CNAME and A records are 60 seconds.

Problem:

AWS ELB IP changes periodically and thats how it works. When the IP changes, Nginx is not picking the new IP address, instead requests are sent to the cached IP (Old IP). This is resulting in 499 after the request_time reaches 59.xxx. 499 is due to ELB closing the request after the 60 seconds Idle time. The issue is Nginx caching the ELB IP and not updating it.

Debugging:

1. I read the docs, checked the previously reported issues. As per my understanding, This issue should not occur. The resolver doc says, By default, Nginx caches answers using the TTL value of a response. My DNS TTL is 60 seconds. I have noticed the error for more than 5 mins, untill nginx is reloaded. Before version 1.1.9, tuning of caching time was not possible, and nginx always cached answers for the duration of 5 minutes..

2. I added the resolver config to override the cache (if there is one).

server {
    listen       81;
    server_name  my.host.name;

    resolver 10.200.0.2 valid=60s;

It's not working. Still noticing that the DNS is cached till reload.

3. Updated the DNS from CNAME to Alias A record. The TTL is always 60 seconds. For CNAME and A records. Still issue exist.

Supporting docs/Logs:

This is a production machine and I have only the logs.

	   log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" "$upstream_addr"'
			' $request_time';

Before Nginx Reload:

10.200.201.145 - - 05/May/2020:19:51:39 +0000 "GET /api/monitor HTTP/1.1" 499 0 "-" "Pingdom.com_bot_version_1.4_http://www.pingdom.com/" "76.72.167.90" "10.200.83.167:80" 59.337

DNS report: Here we can see that the IPs ($upstream_addr) have changed.

# dig +short rails.randomname.us-west-1.infra
internal-a0330ec15812a11e9a1660649a282fad-316863378.us-west-1.elb.amazonaws.com.
10.200.94.115
10.200.94.221

After Nginx reload:

10.200.201.145 - - 05/May/2020:19:52:39 +0000 "GET /api/monitor HTTP/1.1" 200 49 "-" "Pingdom.com_bot_version_1.4_http://www.pingdom.com/" "185.180.12.65" "10.200.94.221:80" 0.003

The issue is always resolved after Nginx reload. Please advise me on how to fix this. Let me know if I can get more data for debugging.

[Maxim Dounin]

Domain names used in nginx configuration are normally resolved during parsing of the configuration. Currently, the only exceptions are:

• When proxy_pass, fastcgi_pass, etc. directives contain variables.
• When using ssl_stapling and resolving OCSP responder hostname.
• When using a server ... resolve; in an upstream block (available as a part of the commercial subscription).

All these cases explicitly document that they are using resolver. For example, quoting ​proxy_pass documentation:

Parameter value can contain variables. In this case, if an address is specified as a domain name, the name is searched among the described server groups, and, if not found, is determined using a ​resolver.

Don't expect that names written in the configuration will be re-resolved by nginx periodically. Depending on the particular use case, you may want to either reconfigure nginx as IP addresses change, or configure nginx in a way which implies period re-resolution of names you want to be re-resolved.

Note that re-resolution implies run-time overhead and also may end up with a non-working nginx if DNS server won't be reachable. On the other hand, nginx reconfiguration implies additional resource usage till reconfiguration complete, which may take a while. For this and other reasons automatic reconfiguration might not be a good idea.