Opened 3 years ago

Last modified 3 years ago

#1151 new enhancement

Use sched_getaffinity() and CPU_COUNT() for ngx_ncpu on Linux

Reported by: nmeyer-otto@… Owned by:
Priority: minor Milestone:
Component: nginx-core Version: 1.11.x
Keywords: Cc:
uname -a: Linux e371d562776f 4.8.0-30-generic #32-Ubuntu SMP Fri Dec 2 03:43:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.11.6
built by gcc 6.2.0 20161005 (Ubuntu 6.2.0-5ubuntu12)
built with OpenSSL 1.1.0c 10 Nov 2016
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -fdebug-prefix-map=/usr/src/nginx-1.11.6=. -fstack-protector-strong -Wformat -Werror=format-security -march=haswell -mtune=haswell -O3 -malign-data=abi' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now'

Description

For better integration with systemd cgroups and containers like docker I humbly suggest using an approach similar to the one employed by nproc from coreutils to determine the number of cpus.

Currently, when using a systemd unit (or docker) and limiting nginx to a few cores, setting worker_processes would result in nginx spawning one cpu per core in the system. I suggest to chagne this to use only the amount of CPUs actually available.

Example:
nginx.conf

worker_processes auto;

/etc/systemd/system/nginx.service.d/cpu-affinity.conf

[Service]
CPUAffinity=0,1

Should only spawn two processes instead of four. The same goes when using docker, e.g.:
docker run --cpuset-cpus 0,1 --rm -t -i -v /etc/nginx/nginx.conf:/etc/nginx/nginx.conf nginx
should spawn 4 workers (not that the default image will only ever start a single worker).

I wrote a small patch that accomplishes this, not being a C coder or familiar with the nginx code base it's very hacky. It does work however.

Attachments (1)

nginx-cgroup-cpu-count.patch (864 bytes ) - added by nmeyer-otto@… 3 years ago.

Download all attachments as: .zip

Change History (3)

by nmeyer-otto@…, 3 years ago

comment:1 by Maxim Dounin, 3 years ago

We've considered something like this previously, though rejected. The problem is that it looks impossible to distinguish between CPUs not set in the mask returned by sched_getaffinity() because they are not available in a particular container, vs. CPUs not set due to previous sched_setaffinity() in the parent process.

comment:2 by Maxim Dounin, 3 years ago

See also #1311.

Note: See TracTickets for help on using tickets.