Opened 5 years ago

Closed 5 years ago

Last modified 5 years ago

#896 closed defect (invalid)

Maxed out DNS connections

Reported by: alex.squarevault.com@… Owned by:
Priority: major Milestone:
Component: nginx-core Version: 1.8.x
Keywords: DNS connections established Cc:
uname -a: Linux server1 3.5.0-51-generic #77~precise1-Ubuntu SMP Thu Jun 5 00:48:28 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.8.0
built with OpenSSL 1.0.1 14 Mar 2012 (running with OpenSSL 1.0.1f 6 Jan 2014)
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_gzip_static_module --without-http_browser_module --without-http_geo_module --without-http_limit_req_module --without-http_limit_zone_module --without-http_memcached_module --without-http_referer_module --without-http_scgi_module --without-http_split_clients_module --without-http_ssi_module --without-http_userid_module --without-http_uwsgi_module --add-module=/build/buildd/nginx-1.8.0/debian/modules/nginx-echo

Description

nginx seems to be establishing and maintaining as many DNS connections as it can.

In my nginx.conf file I have the following lines:


resolver 10.0.80.11 10.0.80.12 8.8.8.8 8.8.4.4 valid=300s;
  resolver_timeout 5s;

Here is a command that lets you see how many connection to an IP your server currently has open that are "ESTABLISHED".

watch -n 0 "netstat -ntu | grep 'ESTABLISHED' | awk '{print \$5}' | cut -d: -f1 | sort | uniq -c | sort -n -r"

When I run this command I get the following results:

http://i.imgur.com/c7Q2RKi.png

As you can see, there are 80 connection to all of my defined namespace resolvers. That number never goes any higher.
On my fresh restart of the nginx process, the first resolver IP will slowly increase in connections until it hits 80, and then the next will slowly increase in connections until it hits 80 and then the third and then the fourth.
It should probably be noted that my machine has 80 cores. Since that's the only other logical thing with the number 80 on my box, I have to assume that it's related.

I do not know if this is causing a performance issue. I would assume the reason that it's establishing new connections is because the previous ones are unavailable, so this may be a bottleneck for my applications.

Change History (5)

comment:1 by Maxim Dounin, 5 years ago

How many worker processes do you have? Each worker processes is expected to maintain one socket to each DNS server, so assuming you are using 80 worker processes (or worker_processes auto on a 80-core server) numbers look correct.

in reply to:  1 comment:2 by alex.squarevault.com@…, 5 years ago

Replying to mdounin:

How many worker processes do you have? Each worker processes is expected to maintain one socket to each DNS server, so assuming you are using 80 worker processes (or worker_processes auto on a 80-core server) numbers look correct.

Ah!

I do have worker_processes auto in my config file.

I haven't seen this kind of behavior on other webservers (namely lighttpd). What is the advantage to taking this approach?

Sorry for the bug report on something I simply didn't understand :)

comment:3 by maxim, 5 years ago

Hi Alex,

unlike lighttpd, that utilize single-process model, nginx uses multi-process model. There are pros and cons, you can learn more about nginx approach here:

http://aosabook.org/en/nginx.html

(Scroll down to "Workers Model" if you are looking for pros.)

While this document is already a bit outdated it is a still good overview of the nginx architecture.

comment:4 by Maxim Dounin, 5 years ago

Resolution: invalid
Status: newclosed

Closing this, as everything seems to work as expected. Maxim's comment explains why the behaviour is different from lighttpd's.

in reply to:  3 comment:5 by alex.squarevault.com@…, 5 years ago

Replying to maxim:

Hi Alex,

unlike lighttpd, that utilize single-process model, nginx uses multi-process model. There are pros and cons, you can learn more about nginx approach here:

http://aosabook.org/en/nginx.html

(Scroll down to "Workers Model" if you are looking for pros.)

While this document is already a bit outdated it is a still good overview of the nginx architecture.

Hi Maxim,

I was more curious as to why it keeps an established connection open to the DNS than why it's using a multi-process model.

Thanks!

Note: See TracTickets for help on using tickets.