Opened 3 years ago

Last modified 2 years ago

#1151 new enhancement

Use sched_getaffinity() and CPU_COUNT() for ngx_ncpu on Linux

Reported by: nmeyer-otto@… Owned by:
Priority: minor Milestone:
Component: nginx-core Version: 1.11.x
Keywords: Cc:
uname -a: Linux e371d562776f 4.8.0-30-generic #32-Ubuntu SMP Fri Dec 2 03:43:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.11.6 built by gcc 6.2.0 20161005 (Ubuntu 6.2.0-5ubuntu12) built with OpenSSL 1.1.0c 10 Nov 2016 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -fdebug-prefix-map=/usr/src/nginx-1.11.6=. -fstack-protector-strong -Wformat -Werror=format-security -march=haswell -mtune=haswell -O3 -malign-data=abi' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now'

Description

For better integration with systemd cgroups and containers like docker I humbly suggest using an approach similar to the one employed by nproc from coreutils to determine the number of cpus.

Currently, when using a systemd unit (or docker) and limiting nginx to a few cores, setting worker_processes would result in nginx spawning one cpu per core in the system. I suggest to chagne this to use only the amount of CPUs actually available.

Example:
nginx.conf

worker_processes auto;

/etc/systemd/system/nginx.service.d/cpu-affinity.conf

[Service]
CPUAffinity=0,1

Should only spawn two processes instead of four. The same goes when using docker, e.g.:
docker run --cpuset-cpus 0,1 --rm -t -i -v /etc/nginx/nginx.conf:/etc/nginx/nginx.conf nginx
should spawn 4 workers (not that the default image will only ever start a single worker).

I wrote a small patch that accomplishes this, not being a C coder or familiar with the nginx code base it's very hacky. It does work however.

Attachments (1)

nginx-cgroup-cpu-count.patch (864 bytes) - added by nmeyer-otto@… 3 years ago.

Download all attachments as: .zip

Change History (3)

Changed 3 years ago by nmeyer-otto@…

comment:1 Changed 3 years ago by mdounin

We've considered something like this previously, though rejected. The problem is that it looks impossible to distinguish between CPUs not set in the mask returned by sched_getaffinity() because they are not available in a particular container, vs. CPUs not set due to previous sched_setaffinity() in the parent process.

comment:2 Changed 2 years ago by mdounin

See also #1311.

Note: See TracTickets for help on using tickets.