nginx proxy_pass variable DNS resolution not updated when there is another proxy_pass with same domain and without variable

[lkgendev@…]

according to the documentation of nginx dns disccovery, when using proxy_pass with a variable for the domain, the variable should be reresolved upon DNS change of IP
however when there are two proxy_pass locations, one with variable and another without variable for the same domain name, it appears that the proxy_pass with variable does not re-resolves the DNS IP changes of the domain name and stays with the IP at the beginning even if DNS changed the domain IP

To reproduce on Redhat 8

1. install nginx: sudo yum install nginx
2. start local dns that can read /etc/hosts entries: sudo systemctl start systemd-resolved
3. modify the nginx configuration file /etc/nginx/nginx.conf

3a. add to the log_format $upstream_addr e.g.

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" "$upstream_addr"';

3b. add two proxy_pass, one with variable resolver of 127.0.0.53 with valid of 1 second and the other without variable

        location /test1 {
            resolver 127.0.0.53 valid=1s;
            set $testbackend https://test.local;
            proxy_pass $testbackend;
        }

        location /test2 {
            proxy_pass https://test.local;
        }

4. add to /etc/hosts an entry for test.local pointing to 127.0.0.200

127.0.0.200 test.local

5. use dig to check the DNS resolution of test.local: dig @127.0.0.53 test.local

check IP is resolved to 127.0.0.200

6. restart nginx for configuration to take effect: sudo systemctl start nginx
7. use curl to connect to the test1 location: curl localhost:80/test1
8. check nginx access log last lines, there should be upstream address of 127.0.0.200:443

tail -20 /var/log/nginx/access.log
...
::1 - - [04/Apr/2024:15:07:46 +0300] "GET /test1 HTTP/1.1" 502 4020 "-" "curl/7.61.1" "-" "127.0.0.200:443"
upstram IP is 127.0.0.200

9. change the /etc/hosts entry for test.local to have a different IP 127.0.0.201 instead of the previous 127.0.0.200

127.0.0.201 test.local

10. use dig to check the DNS resolution of test.local: dig @127.0.0.53 test.local

check IP is resolved to 127.0.0.201

11. wait for 30 seconds so the valid time for the DNS entry should expire for the resolver
12. use curl to connect to the test1 location: curl localhost:80/test1
13. check nginx access log last lines, the upstream address should change to 127.0.0.201:443 but it will remain as before

tail -20 /var/log/nginx/access.log
...
::1 - - [04/Apr/2024:15:17:27 +0300] "GET /test1 HTTP/1.1" 502 4020 "-" "curl/7.61.1" "-" "127.0.0.200:443"

[lkgendev@…]

nginx configuration to reproduce the DNS resolution problem with domain name test.local

[lkgendev@…]

pasted the later access log line

[Roman Arutyunyan]

When using variables in proxy_pass, nginx always tries to find an existing upstream by the name evaluated in runtime. If the upstream exists, it's used, otherwise DNS resolve is performed and a temporary upstream is created on the fly. In your case such an upstream always exists since it's created in config-time by the second proxy_pass.

Also, editing /etc/hosts would not help anyway since nginx does not read this file in runtime. Instead, when resolving names in runtime, it sends a request directly to the resolver. In contrast, when resolving names in config-time, nginx uses libc resolve functions which do read /etc/hosts.

[lkgendev@…]

According to the documentation, the resolver valid keyword specifies that variables need to re-resolve the DNS name after the time set in valid
"
When you use a variable to specify the domain name in the proxy_pass directive, NGINX re‑resolves the domain name when its TTL expires. You must include the resolver directive to explicitly specify the name server (NGINX does not refer to /etc/resolv.conf as in the first two methods). By including the valid parameter to the resolver directive, you can tell NGINX to ignore the TTL and re‑resolve names at a specified frequency instead. Here we tell NGINX to re‑resolve names every 10 seconds.
"
The same issue will happen also with a regular DNS so it is not /etc/hosts related, using the 127.0.0.53 as a local DNS was used just to simplify the use case reproduction

[Roman Arutyunyan]

When an upstream with the evaluated name is found, no resolve happens.

[lkgendev@…]

Thank you for clarifying the proxy_pass domain resolution order