Opened 4 years ago

Closed 4 years ago

#974 closed enhancement (invalid)

Resolver fails if IPv6 DNS records are not returned

Reported by: Kevin Decker Owned by:
Priority: minor Milestone:
Component: other Version: 1.9.x
Keywords: Cc:
uname -a: Linux nginx.weave.local 4.4.5-15.26.amzn1.x86_64 #1 SMP Wed Mar 16 17:15:34 UTC 2016 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.9.15
built by gcc 4.9.2 (Debian 4.9.2-10)
built with OpenSSL 1.0.1k 8 Jan 2015
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_perl_module=dynamic --add-dynamic-module=debian/extra/njs-1c50334fbea6/nginx --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --with-http_v2_module --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,--as-needed'

Description

Attempting to utilize Weave DNS to do container resolution. When using the startup-based resolution and upstream definitions, the server would launch successfully with all of the standard caveats around existing at startup, etc.

upstream staging {
  server server-staging.weave.local:3000;
}

server {
  location @proxy {
    proxy_pass http://staging;
  }
}

Unfortunately when trying to do the same setup using the resolver, this would not resolve for these instances, but would for global such a google.com.

resolver 172.17.0.1:53;

server {
  location @proxy {
    set $staging "server-staging.weave.local:3000";
    proxy_pass http://$staging;
  }
}

When enabling error logs, the only useful output was:

2016/05/05 08:42:56 [error] 14#14: *1 server-staging.weave.local could not be resolved (3: Host not found), client: private.ip, server: public.server, request: "GET / HTTP/1.1", host: "public.server"

Eventually, by looking at the full debug logs, I was able to determine that the issue was this record did not have a IPv6 entry.

 2016/05/05 08:42:56 [debug] 14#14: resolve: "server-staging.weave.local"
 2016/05/05 08:42:56 [debug] 14#14: resolve: "server-staging.weave.local" A 53493
 2016/05/05 08:42:56 [debug] 14#14: resolve: "server-staging.weave.local" AAAA 12359
 2016/05/05 08:42:56 [debug] 14#14: resolver DNS response 53493 fl:8580 1/1/0/0
 2016/05/05 08:42:56 [debug] 14#14: resolver DNS response qt:1 cl:1
 2016/05/05 08:42:56 [debug] 14#14: resolver qs:server-staging.weave.local
 2016/05/05 08:42:56 [debug] 14#14: resolver naddrs:1 cname:0000000000000000 ttl:1
 2016/05/05 08:42:56 [debug] 14#14: resolver DNS response 12359 fl:8183 1/0/0/0
 2016/05/05 08:42:56 [debug] 14#14: resolver DNS response qt:28 cl:1
 2016/05/05 08:42:56 [debug] 14#14: resolver qs:server-staging.weave.local
 2016/05/05 08:42:56 [debug] 14#14: *1 http upstream resolve: "/?"
 2016/05/05 08:42:56 [error] 14#14: *1 server-staging.weave.local could not be resolved (3: Host not found), client: private.ip, server: public.ip, request: "GET / HTTP/1.1", host: "public.ip"

vs. google.com:

 2016/05/05 08:39:36 [debug] 10#10: resolve: "google.com"
 2016/05/05 08:39:36 [debug] 10#10: resolve: "google.com" A 42052
 2016/05/05 08:39:36 [debug] 10#10: resolve: "google.com" AAAA 26761
 2016/05/05 08:39:36 [debug] 10#10: resolver DNS response 26761 fl:8180 1/1/0/0
 2016/05/05 08:39:36 [debug] 10#10: resolver DNS response qt:28 cl:1
 2016/05/05 08:39:36 [debug] 10#10: resolver qs:google.com
 2016/05/05 08:39:36 [debug] 10#10: resolver naddrs:1 cname:0000000000000000 ttl:293
 2016/05/05 08:39:36 [debug] 10#10: resolver DNS response 42052 fl:8180 1/1/0/0
 2016/05/05 08:39:36 [debug] 10#10: resolver DNS response qt:1 cl:1
 2016/05/05 08:39:36 [debug] 10#10: resolver qs:google.com
 2016/05/05 08:39:36 [debug] 10#10: resolver naddrs:1 cname:0000000000000000 ttl:156
 2016/05/05 08:39:36 [debug] 10#10: *1 http upstream resolve: "/?"
 2016/05/05 08:39:36 [debug] 10#10: *1 name was resolved to 172.217.1.14
 2016/05/05 08:39:36 [debug] 10#10: *1 name was resolved to 2607:f8b0:4004:809::200e

I have little experience with IPv6 at this point, so I'm not sure what the proper behavior is here. Ideally these records would not be rejected, but if the best practice is for them to be, it would be a huge time saver if the error logging would be expanded to state that the rejection was due to a partial result.

Workaround/Fix for end user: Add ipv6=off to Weave resolver definition.

resolver 172.17.0.1:53 ipv6=off;

Change History (3)

comment:1 by Maxim Dounin, 4 years ago

The problem is the response says that the domain name does not exists, and that's exactly what nginx writes to the logs. If the name exists, but there are no AAAA records - your DNS server is expected to return empty responses for AAAA queries instead, with no errors set. And that's what normal DNS servers do. You may consider reporting this to Weave DNS developers, they have a bug.

comment:2 by Kevin Decker, 4 years ago

I reported this indirectly and they pointed me to this as potentially related:
https://github.com/weaveworks/weave/issues/2244

I am not familiar enough with the DNS protocols to comment on what the correct behavior is. Logging some sort of "your DNS server is inconsistent" error would be immensely helpful if the server is non-compliant, which it is sounding like might be the case.

comment:3 by Ruslan Ermilov, 4 years ago

Resolution: invalid
Status: newclosed

Not an nginx bug which was already fixed.

Note: See TracTickets for help on using tickets.