Opened 7 years ago

Last modified 3 years ago

#1535 new enhancement

proxy_bind and resolver IP version mismatch

Reported by: zingaburga@… Owned by:
Priority: minor Milestone:
Component: other Version: 1.13.x
Keywords: proxy_bind, resolver Cc:
uname -a: Linux 4.9.0-3-amd64
nginx -V: 1.13.3

Description

If proxy_bind is used and we're proxying to a hostname which lists both IPv4 and IPv6 addresses, the request can randomly fail depending on what address the resolver decides to pick.
This can be confusing to diagnose (and makes proxy_bind look like it's broken) since the proxy can appear to work, but rather unreliably.

I found the following which states there is no workaround to the problem: http://nginx.2469901.n2.nabble.com/How-Nginx-behaves-with-quot-proxy-bind-quot-and-DNS-resolver-with-non-matching-ip-versions-between-b-td7592529.html

I'm still seeing this problem on v1.13.3 so it doesn't seem to have been resolved yet.

Most domains which have an IPv6 address will also list an IPv4 address. This is particularly problematic if we're binding on an IPv6 address because there's also no way to force the resolver to only give IPv6: https://forum.nginx.org/read.php?10,270086

I can't think of any reason why you'd ever want the bind and upstream to not be on the same IP version (otherwise it's guaranteed to fail), so it makes a lot of sense if this could be addressed.


I also did try forcing everything to IPv4, but it didn't seem to work for me - maybe I've made a mistake somewhere? Config looks like:

stream { server {
  listen 5555;
  resolver 1.1.1.1 ipv6=off;
  proxy_pass example.com:80;
  proxy_bind 0.0.0.0;
}}

Executing nc 0 5555, every now and then, it fails and I get the following in the error log:

2018/04/19 13:51:08 [crit] 18368#18368: *330 bind(0.0.0.0) failed (22: Invalid argument) while connecting to upstream, client: 127.0.0.1, server: 0.0.0.0:5555, upstream: "[2606:2800:220:1:248:1893:25c8:1946]:80", bytes from/to client:0/0, bytes from/to upstream:0/0

Change History (6)

comment:1 by Maxim Dounin, 7 years ago

I can't think of any reason why you'd ever want the bind and upstream to not be on the same IP version (otherwise it's guaranteed to fail)

Most simple solution would be to remove proxy_bind from such configuration. The proxy_bind directive exists to address cases when you have to ensure that a particular address will be used in outgoing connections, and it is expected to fail if the address cannot be used.

I also did try forcing everything to IPv4, but it didn't seem to work for me - maybe I've made a mistake somewhere?

In the example provided, the name "example.com" as written in proxy_pass will be resolved during configuration parsing, using the getaddrinfo() call, that is, using the system resolver. If you want to use only IPv4 addresses of example.com, you have to configure your resolver accordingly.

comment:2 by zingaburga@…, 7 years ago

Thanks for the response.

Most simple solution would be to remove proxy_bind from such configuration

Well... I suppose that works but now you can't force a particular local address to be used...

The proxy_bind directive exists to address cases when you have to ensure that a particular address will be used in outgoing connections, and it is expected to fail if the address cannot be used.

So if you're binding to an IPv6 address, wouldn't it make sense that you only ever want to connect to an upstream via IPv6? It makes no sense to attempt a connection via IPv4!

that is, using the system resolver

Oh, so the resolver configuration option doesn't apply in this case? That's confusing :/
Changing the system resolver for just this seems like an overkill solution anyway (and may be undesirable, e.g. if you do want IPv6 resolution for other applications).

comment:3 by poige@…, 5 years ago

Hi! Adding support for several IPs would solve the whole issue. When binding, the list of specified IPs should be searched to find the matching ones. For e. g., that block would restrict IPv4 only and leave IPv6 unbound.

proxy_bind 1.2.3.4 [::];

In case the list of bind-IPs doesn't have explicitly given IPv6 addresses Nginx should fail instead of using [::] implicitly.

At least it looks more reasonable compared to additional proxy_bind_v6 or alike.

comment:4 by https://stackoverflow.com/users/9252236/mayank-j, 5 years ago

Hi!

Been there done that !

proxy_bind does randomly behave weird with IPv6.

I think I've resolved this issue now.
Was desperate for the solution and this thread gives me the heads-up where to start looking from.

Kindly add below 3 properties in sysctl & try running again.

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

Last edited 5 years ago by https://stackoverflow.com/users/9252236/mayank-j (previous) (diff)

in reply to:  4 comment:5 by poige@…, 5 years ago

Replying to https://stackoverflow.com/users/9252236/mayank-j:

Hi!

Been there done that !

Thrown the baby out with the bathwater is what you've done.

Of course it's not a solution to get rid of IPv6 in 2020.

comment:6 by Maxim Dounin, 3 years ago

See also #2308.

Note: See TracTickets for help on using tickets.