Opened 8 years ago
Closed 8 years ago
#1248 closed defect (invalid)
syslog UDP sockets growing until 28230 connections.
Reported by: | Owned by: | ||
---|---|---|---|
Priority: | minor | Milestone: | |
Component: | nginx-core | Version: | 1.8.x |
Keywords: | syslog connections saturation | Cc: | |
uname -a: | Linux pepitaweb11 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18 16:48:51 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | ||
nginx -V: |
nginx version: nginx/1.8.0
built with OpenSSL 1.0.1f 6 Jan 2014 TLS SNI support enabled configure arguments: --prefix=/opt/nginx-1.8.0 --user=www-data --group=www-data --with-http_gzip_static_module --with-http_gunzip_module --with-http_ssl_module --with-http_spdy_module --with-http_flv_module --with-http_mp4_module --with-http_secure_link_module --with-http_dav_module --with-http_realip_module --with-file-aio --with-debug --with-http_stub_status_module --with-pcre --with-pcre-jit --with-pcre=/opt/arc/build/pcre-8.36 |
Description
On Linux server with large number of traffic, using syslog for remote logging output cause UDP syslog connections to grow until 28230 connections after few days and then block any other new UDP connections.
system is Ubuntu Trusty (14.04), with self compiled nginx 1.8.0 remote logging use this kind of configuration in several virtual hosts
access_log syslog:server=10.99.0.47,facility=local1,severity=info main; error_log syslog:server=10.99.0.47,facility=local1,severity=error notice;
for counting the number of connection and 28230 limit:
netstat -au | grep ESTABLISHED | wc -l
the accumulated connections are of this kind (using netstat -aun):
udp 0 0 10.99.10.16:51676 10.99.0.47:514 ESTABLISHED
They keep the state ESTABLISHED
Default net.core.somaxconn = 30000
is used on this system. After nginx has reached 28230 connections the only solution to resolve the issue is to restart nginx to free connections.
Change History (3)
comment:1 by , 8 years ago
comment:2 by , 8 years ago
In this configuraiton, there are 1024 workers to manage burst usage.
worker_processes 1024;
There is about 50 vhosts, with for each vhost, the following logging rules:
access_log syslog:server=10.99.0.47,facility=local1,severity=info main;
error_log syslog:server=10.99.0.47,facility=local1,severity=error notice;
This could do 50 * 1024 = 51200 UDP connections, and then could be problem? Having a one time only global logging rules could perhaps avoid this?
comment:3 by , 8 years ago
Resolution: | → invalid |
---|---|
Status: | new → closed |
Ok, this perfectly explains observed behaviour. With 1024 worker processes, 50 servers, and 2 logs in each server it is expected that up to 1024 * 50 * 2 == 102400 sockets can be used. Using global logs at http
level should improve things.
Note well that using 1024 worker processes "to manage burst usage" looks wrong. In nginx, each worker process can handle thousands of connections from clients. For effective operation, number of worker processes should be set according system resources to be used by nginx - usually CPU cores or hard disks.
First of all you may want to check if you see the problem with a less ancient nginx version. Current stable version is 1.12.0, and nginx 1.8.0 is not supported for more than a year now.
It would be also a good idea to show the actual configuration which demonstrates the problem. Or, at least, to define "several virtual hosts". Note that each
access_log syslog:...
and/orerror_log syslog:...
in the configuration is expected to create a socket.