syslog UDP sockets growing until 28230 connections.

[olivier.gondouin@…]

On Linux server with large number of traffic, using syslog for remote logging output cause UDP syslog connections to grow until 28230 connections after few days and then block any other new UDP connections.

system is Ubuntu Trusty (14.04), with self compiled nginx 1.8.0 remote logging use this kind of configuration in several virtual hosts

access_log syslog:server=10.99.0.47,facility=local1,severity=info main;
error_log syslog:server=10.99.0.47,facility=local1,severity=error notice;

for counting the number of connection and 28230 limit:

  netstat -au | grep ESTABLISHED | wc -l

the accumulated connections are of this kind (using netstat -aun):

udp        0      0 10.99.10.16:51676        10.99.0.47:514          ESTABLISHED

They keep the state ESTABLISHED

Default net.core.somaxconn = 30000 is used on this system. After nginx has reached 28230 connections the only solution to resolve the issue is to restart nginx to free connections.

[Maxim Dounin]

First of all you may want to check if you see the problem with a less ancient nginx version. Current stable version is 1.12.0, and nginx 1.8.0 is not supported for more than a year now.

It would be also a good idea to show the actual configuration which demonstrates the problem. Or, at least, to define "several virtual hosts". Note that each access_log syslog:... and/or error_log syslog:... in the configuration is expected to create a socket.

[olivier.gondouin@…]

In this configuraiton, there are 1024 workers to manage burst usage.

worker_processes 1024;

There is about 50 vhosts, with for each vhost, the following logging rules:

access_log syslog:server=10.99.0.47,facility=local1,severity=info main;
error_log syslog:server=10.99.0.47,facility=local1,severity=error notice;

This could do 50 * 1024 = 51200 UDP connections, and then could be problem? Having a one time only global logging rules could perhaps avoid this?

[Maxim Dounin]

Ok, this perfectly explains observed behaviour. With 1024 worker processes, 50 servers, and 2 logs in each server it is expected that up to 1024 * 50 * 2 == 102400 sockets can be used. Using global logs at http level should improve things.

Note well that using 1024 worker processes "to manage burst usage" looks wrong. In nginx, each worker process can handle thousands of connections from clients. For effective operation, number of worker processes should be set according system resources to be used by nginx - usually CPU cores or hard disks.