Specific IP(v4) listen directives cause "address already in use" when using ipv6only=off

[indecisivemal@…]

Attempting to have one server block listening on all addresses (v4 and v6) and a second server block listening only on one IPv4 address.

When using ipv6only=off to accomplish the first with a single listen statement, the specific IP listen statement in the second causes nginx to report "address already in use" and exit(1):

Dec 15 01:15:35 mydebian systemd[1]: Starting nginx - high performance web server...
Dec 15 01:15:35 mydebian nginx[20599]: nginx: [emerg] bind() to 127.0.0.1:80 failed (98: Address already in use)
[previous message repeated 4 more times]
Dec 15 01:15:37 mydebian nginx[20599]: nginx: [emerg] still could not bind()
Dec 15 01:15:37 mydebian systemd[1]: nginx.service: Control process exited, code=exited status=1

Scattering log calls around, it seems like src/http/ngx_http.c ngx_http_add_listen() needs awareness of ipv6only=off so when it tests new proto=AF_INET :80 against existing proto=AF_INET6 :80 it considers it known instead of new.

Example nginx.conf:

events {}
http {
    server {
        listen [::]:80 ipv6only=off;
    }
    server {
        listen 127.0.0.1:80;
    }
}

Problem observed with:
Arch nginx-mainline 1.13.7 (Linux myarch 4.14.5-1-ARCH #1 SMP PREEMPT Sun Dec 10 14:50:30 UTC 2017 x86_64 GNU/Linux)
Arch with nginx 1.13.7 from hg, default ./configure
Debian with nginx-mainline from nginx repo (shown)

Both default to net.ipv6.bindv6only=0

[Maxim Dounin]

The problem here is Linux-specific "security" check which doesn't allow listening sockets to coexists with more specific ones on the same port, like 0.0.0.0:80 and 127.0.0.1:80. While nginx handles this when you specify listening sockets without additional options, it is trivial to run into problems on Linux if you specify additional parameters of the listen directive. Quoting the description of the bind parameter of the ​listen directive:

bind

instructs to make a separate bind() call for a given address:port pair. This is useful because if there are several listen directives with the same port but different addresses, and one of the listen directives listens on all addresses for the given port (*:port), nginx will bind() only to *:port. It should be noted that the getsockname() system call will be made in this case to determine the address that accepted the connection. If the setfib, backlog, rcvbuf, sndbuf, accept_filter, deferred, ipv6only, or so_keepalive parameters are used then for a given address:port pair a separate bind() call will always be made.

There are no plans to teach nginx to recognize IPv6-with-mapped-IPv4 listening sockets and merge them with IPv4 sockets on the same port like we do with wildcard and non-wildcard addresses. In particular, because the configuration provided will result in IPv6 connections in the server with listen 127.0.0.1:80, which is at least unexpected (and completely broken in some more complex configrations - for example, when the client address is used in proxy_bind).

Instead, nginx universally switches on the IPV6_V6ONLY socket option by default (see bdcdbdf35b52). To support both IPv6 and IPv4, configure listening sockets for both protocols:

listen 80;
listen [::]:80;

This way nginx will be able to work with listening sockets in a predictable way, identical on a wide range of platforms. And it will be able to properly merge wildcard and non-wildcard listening sockets to avoid problems with the Linux-specific check in question.

If you still want to configure IPv6-with-IPv4 sockets instead for some reasons, you can do so, but it is your responsibility to avoid configurations not allowed by your OS.