strange "worker_connections are not enough, reusing connections" warning

[gabriel.dodan@…]

Hi,

I keep getting that warning, although according to nginx configuration i shouldn't. This is my configuration

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


worker_rlimit_nofile 120000;
events {
    worker_connections  10000;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

The server has 12 cores , so according to nginx configuration the nginx server should handle up to 120000 connections. I have activated stub_status and it shows 6190 active connections . So I should no get that warning because active connections are way less than what nginx is configured to handle.

this is the exact warning i get

2021/11/26 13:50:23 [warn] 114001#114001: 10000 worker_connections are not enough, reusing connections

I use nginx mainly as proxy, proxied servers are websocket servers

Any idea?

Thanks and regards!

[Maxim Dounin]

The stub_status counters show amount of client connections, while worker_connections are also used for listening sockets and, most notably, for upstream connections. The message suggests that one of your worker processes used all or almost all of its 10000 worker_connections, and started to reuse keepalive connections.

Given the total number of worker processes, this suggests poor distribution of client connections between worker processes. Recent reports suggest that it might be bad with recent Linux kernels when using default settings, see ​this thread. There is an ongoing work to further analyse and improve the default configuration. Meanwhile, configuring ​accept_mutex on;, or using listen ... reuseport, or recompiling nginx with --with-cc-opt="-DNGX_HAVE_EPOLLEXCLUSIVE=0" should improve the distribution.

[gabriel.dodan@…]

Thank you very much for the reply! I have turned on the accept_mutex , will watch what happens.

Is there any way to find how many connections are open on each worker process?

Regards!

[Maxim Dounin]

Is there any way to find how many connections are open on each worker process?

Sure, by using normal system tools. On Linux, you can do so by using ss -ntp under root, it will print corresponding processes for each socket, or by using something like lsof -n -i (optionally with -p <pid> to see sockets from a specific worker process).

[gabriel.dodan@…]

Works fine so far, no more warning. Thanks for the suggestion!

Best!

[Maxim Dounin]

Thanks for the feedback. Keeping this open for now, to make sure the underlying problem is not forgotten.

Just in case, it would be great to check if recompiling with --with-cc-opt="-DNGX_HAVE_EPOLLEXCLUSIVE=0" also fixes things in your case, without using accept_mutex on;. If it does, this is probably what we should do by default.

[Maxim Dounin <mdounin@…>]

In 7992:e2d07e4ec636/nginx:

Events: fixed balancing between workers with EPOLLEXCLUSIVE.

Linux with EPOLLEXCLUSIVE usually notifies only the process which was first
to add the listening socket to the epoll instance. As a result most of the
connections are handled by the first worker process (ticket #2285). To fix
this, we re-add the socket periodically, so other workers will get a chance
to accept connections.

[Maxim Dounin]

Underlying problem fixed in e2d07e4ec636, distribution of client connections between worker processes should be even with the default settings.