#896 closed defect (invalid)
Maxed out DNS connections
| Reported by: | Owned by: | ||
|---|---|---|---|
| Priority: | major | Milestone: | |
| Component: | nginx-core | Version: | 1.8.x |
| Keywords: | DNS connections established | Cc: | |
| uname -a: | Linux server1 3.5.0-51-generic #77~precise1-Ubuntu SMP Thu Jun 5 00:48:28 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | ||
| nginx -V: |
nginx version: nginx/1.8.0
built with OpenSSL 1.0.1 14 Mar 2012 (running with OpenSSL 1.0.1f 6 Jan 2014) TLS SNI support enabled configure arguments: --with-cc-opt='-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_gzip_static_module --without-http_browser_module --without-http_geo_module --without-http_limit_req_module --without-http_limit_zone_module --without-http_memcached_module --without-http_referer_module --without-http_scgi_module --without-http_split_clients_module --without-http_ssi_module --without-http_userid_module --without-http_uwsgi_module --add-module=/build/buildd/nginx-1.8.0/debian/modules/nginx-echo |
||
Description
nginx seems to be establishing and maintaining as many DNS connections as it can.
In my nginx.conf file I have the following lines:
resolver 10.0.80.11 10.0.80.12 8.8.8.8 8.8.4.4 valid=300s; resolver_timeout 5s;
Here is a command that lets you see how many connection to an IP your server currently has open that are "ESTABLISHED".
watch -n 0 "netstat -ntu | grep 'ESTABLISHED' | awk '{print \$5}' | cut -d: -f1 | sort | uniq -c | sort -n -r"
When I run this command I get the following results:
As you can see, there are 80 connection to all of my defined namespace resolvers. That number never goes any higher.
On my fresh restart of the nginx process, the first resolver IP will slowly increase in connections until it hits 80, and then the next will slowly increase in connections until it hits 80 and then the third and then the fourth.
It should probably be noted that my machine has 80 cores. Since that's the only other logical thing with the number 80 on my box, I have to assume that it's related.
I do not know if this is causing a performance issue. I would assume the reason that it's establishing new connections is because the previous ones are unavailable, so this may be a bottleneck for my applications.
Change History (5)
follow-up: 2 comment:1 by , 8 years ago
comment:2 by , 8 years ago
Replying to mdounin:
How many worker processes do you have? Each worker processes is expected to maintain one socket to each DNS server, so assuming you are using 80 worker processes (or
worker_processes autoon a 80-core server) numbers look correct.
Ah!
I do have worker_processes auto in my config file.
I haven't seen this kind of behavior on other webservers (namely lighttpd). What is the advantage to taking this approach?
Sorry for the bug report on something I simply didn't understand :)
follow-up: 5 comment:3 by , 8 years ago
Hi Alex,
unlike lighttpd, that utilize single-process model, nginx uses multi-process model. There are pros and cons, you can learn more about nginx approach here:
http://aosabook.org/en/nginx.html
(Scroll down to "Workers Model" if you are looking for pros.)
While this document is already a bit outdated it is a still good overview of the nginx architecture.
comment:4 by , 8 years ago
| Resolution: | → invalid |
|---|---|
| Status: | new → closed |
Closing this, as everything seems to work as expected. Maxim's comment explains why the behaviour is different from lighttpd's.
comment:5 by , 8 years ago
Replying to maxim:
Hi Alex,
unlike lighttpd, that utilize single-process model, nginx uses multi-process model. There are pros and cons, you can learn more about nginx approach here:
http://aosabook.org/en/nginx.html
(Scroll down to "Workers Model" if you are looking for pros.)
While this document is already a bit outdated it is a still good overview of the nginx architecture.
Hi Maxim,
I was more curious as to why it keeps an established connection open to the DNS than why it's using a multi-process model.
Thanks!
How many worker processes do you have? Each worker processes is expected to maintain one socket to each DNS server, so assuming you are using 80 worker processes (or
worker_processes autoon a 80-core server) numbers look correct.