Opened 7 weeks ago

Last modified 7 weeks ago

#2625 new defect

nginx proxy_pass variable DNS resolution not updated when there is another proxy_pass with same domain and without variable

Reported by: lkgendev@… Owned by:
Priority: minor Milestone:
Component: nginx-module Version: 1.14.x
Keywords: proxy_pass dns Cc: lkgendev@…
uname -a: Linux dba-tlv-wkrlxy 4.18.0-477.10.1.el8_8.x86_64 #1 SMP Wed Apr 5 13:35:01 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.14.1
built by gcc 8.2.1 20180905 (Red Hat 8.2.1-3) (GCC)
built with OpenSSL 1.1.1 FIPS 11 Sep 2018 (running with OpenSSL 1.1.1k FIPS 25 Mar 2021)
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-file-aio --with-ipv6 --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_slice_module --with-http_stub_status_module --with-http_perl_module=dynamic --with-http_auth_request_module --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-debug --with-cc-opt='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E'

Description (last modified by lkgendev@…)

according to the documentation of nginx dns disccovery, when using proxy_pass with a variable for the domain, the variable should be reresolved upon DNS change of IP
however when there are two proxy_pass locations, one with variable and another without variable for the same domain name, it appears that the proxy_pass with variable does not re-resolves the DNS IP changes of the domain name and stays with the IP at the beginning even if DNS changed the domain IP

To reproduce on Redhat 8

  1. install nginx: sudo yum install nginx
  2. start local dns that can read /etc/hosts entries: sudo systemctl start systemd-resolved
  3. modify the nginx configuration file /etc/nginx/nginx.conf

3a. add to the log_format $upstream_addr e.g.

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" "$upstream_addr"';

3b. add two proxy_pass, one with variable resolver of 127.0.0.53 with valid of 1 second and the other without variable

        location /test1 {
            resolver 127.0.0.53 valid=1s;
            set $testbackend https://test.local;
            proxy_pass $testbackend;
        }

        location /test2 {
            proxy_pass https://test.local;
        }
  1. add to /etc/hosts an entry for test.local pointing to 127.0.0.200

127.0.0.200 test.local

  1. use dig to check the DNS resolution of test.local: dig @127.0.0.53 test.local

check IP is resolved to 127.0.0.200

  1. restart nginx for configuration to take effect: sudo systemctl start nginx
  2. use curl to connect to the test1 location: curl localhost:80/test1
  3. check nginx access log last lines, there should be upstream address of 127.0.0.200:443

tail -20 /var/log/nginx/access.log
...
::1 - - [04/Apr/2024:15:07:46 +0300] "GET /test1 HTTP/1.1" 502 4020 "-" "curl/7.61.1" "-" "127.0.0.200:443"
upstram IP is 127.0.0.200

  1. change the /etc/hosts entry for test.local to have a different IP 127.0.0.201 instead of the previous 127.0.0.200

127.0.0.201 test.local

  1. use dig to check the DNS resolution of test.local: dig @127.0.0.53 test.local

check IP is resolved to 127.0.0.201

  1. wait for 30 seconds so the valid time for the DNS entry should expire for the resolver
  2. use curl to connect to the test1 location: curl localhost:80/test1
  3. check nginx access log last lines, the upstream address should change to 127.0.0.201:443 but it will remain as before

tail -20 /var/log/nginx/access.log
...
::1 - - [04/Apr/2024:15:17:27 +0300] "GET /test1 HTTP/1.1" 502 4020 "-" "curl/7.61.1" "-" "127.0.0.200:443"

Attachments (1)

nginx.conf (2.7 KB ) - added by lkgendev@… 7 weeks ago.
nginx configuration to reproduce the DNS resolution problem with domain name test.local

Download all attachments as: .zip

Change History (7)

by lkgendev@…, 7 weeks ago

Attachment: nginx.conf added

nginx configuration to reproduce the DNS resolution problem with domain name test.local

comment:1 by lkgendev@…, 7 weeks ago

Description: modified (diff)

comment:2 by lkgendev@…, 7 weeks ago

pasted the later access log line

comment:3 by Roman Arutyunyan, 7 weeks ago

When using variables in proxy_pass, nginx always tries to find an existing upstream by the name evaluated in runtime. If the upstream exists, it's used, otherwise DNS resolve is performed and a temporary upstream is created on the fly. In your case such an upstream always exists since it's created in config-time by the second proxy_pass.

Also, editing /etc/hosts would not help anyway since nginx does not read this file in runtime. Instead, when resolving names in runtime, it sends a request directly to the resolver. In contrast, when resolving names in config-time, nginx uses libc resolve functions which do read /etc/hosts.

comment:4 by lkgendev@…, 7 weeks ago

According to the documentation, the resolver valid keyword specifies that variables need to re-resolve the DNS name after the time set in valid
"
When you use a variable to specify the domain name in the proxy_pass directive, NGINX re‑resolves the domain name when its TTL expires. You must include the resolver directive to explicitly specify the name server (NGINX does not refer to /etc/resolv.conf as in the first two methods). By including the valid parameter to the resolver directive, you can tell NGINX to ignore the TTL and re‑resolve names at a specified frequency instead. Here we tell NGINX to re‑resolve names every 10 seconds.
"
The same issue will happen also with a regular DNS so it is not /etc/hosts related, using the 127.0.0.53 as a local DNS was used just to simplify the use case reproduction

comment:5 by Roman Arutyunyan, 7 weeks ago

When an upstream with the evaluated name is found, no resolve happens.

Last edited 7 weeks ago by Roman Arutyunyan (previous) (diff)

comment:6 by lkgendev@…, 7 weeks ago

Thank you for clarifying the proxy_pass domain resolution order

Note: See TracTickets for help on using tickets.