Add optional systemd socket activation support

[David Strauss]

The systemd project supports socket activation for services, allowing systemd to listen on the socket initially. When the first connection comes in, systemd starts the service and passes in any listening sockets as file descriptors.

The following patch, which is ready for general implementation review, adds support with a --with-system configuration option. When this support is compiled in, nginx can use "listen fd:3" (and other numbers) to specify that a given nginx server should use the inherited file descriptor in lieu of opening its own socket.

The implementation initializes nginx's data structures with (mostly) the same information that would generally be derived from the configured listener. For example, nginx will know the listening IP address and port of an INET or INET6 socket.

Why nginx benefits from this:

• systemd can listen on privileged ports or socket paths and start nginx with privileges already dropped. This is good for shared/multitenant environments where user configuration of nginx is offered.
• Virtually no memory or CPU is in use until nginx receives the first request. There is a ~20ms overhead for the first request while nginx starts. It is also possible with systemd socket activation to start nginx by default, before any request has come in.
• Any service in front of nginx doesn't have to wait for nginx to finish starting (or start at all) before sending in the first request. This is useful for setups where something like Varnish is in front of nginx.
• Major upgrades or reconfigurations of nginx requiring a full restart are possible without having the listening socket(s) ever go away. If nginx does a clean shutdown (without killing any requests), it's possible to have zero requests fail and no service interruption from the nginx restart.

Known TODOs or possible changes needed in the implementation:

• Obviously, logging and printf calls need cleanup.
• Unix sockets don't initialize the path as the address name. They list the "fd:N" string. This doesn't affect functionality, but it might be nice to fix.
• No documentation updates made yet.

[David Strauss]

Also, #ifdef for Unix domain socket support may be redundant in a block of code exclusively related to systemd.

[Maxim Dounin]

For binary upgrade on the fly (as available from very start, see docs ​here) nginx is capable of using inherited file descriptors for listening sockets, via NGINX environment variable with a list of file descriptors to use. You may also try to reuse the same interface for some external control. But I don't think that it would be good idea to make this directly configurable by users.

Standard disclaimer: using trac to submit patches is bad idea, use ​nginx-devel@ mailing list instead.

[David Strauss]

Yes, I realize nginx supports its own mechanism for binary upgrades. systemd socket activation goes a bit beyond this, allowing configuration management to safely stop nginx as part of configuration management. We pack 500+ isolated instances on each machine and leave most stopped and ready for socket activation. When we reconfigure nginx, we simply cleanly stop each instance after reconfiguration to avoid launching hundreds of formerly idle instances.

It may be possible to do this as a hack by carefully using internal nginx's internal, undocumented environmental variables, but this is the wrong place for integration, as I'll explain.

We're moving in Fedora (and, implicitly, RHEL) to shipping more services with socket activation as the primary way listeners get configured and managed. We shouldn't have to use undocumented internals of nginx in the systemd service units we ship. In fact, it's likely Fedora/RHEL won't ship socket activation units for nginx without there being a documented way to pass in existing file descriptors for sockets.

It's also a poor user experience. Without a deep knowledge of the nginx source code, it's unclear how to properly configure a systemd socket, a systemd service, and an nginx configuration file to work together. And, even when you do, you end up with either the "listen" directive redundantly specifying the same listener as the systemd socket unit or being silently ignored entirely.

In summary, some major benefits remain:

• More efficient instance management for configuration changes (clean shutdown vs. reload).
• Fedora can ship socket activation for nginx without using undocumented features of nginx.
• Users can configure their own corresponding systemd sockets and nginx instances without source code knowledge and with a clear mapping from the sockets in systemd to the servers in nginx.

[David Strauss]

Reopening because I still think this merits discussion.

[David Strauss]

Another benefit: systemd sockets support deep integration with selinux- and SMACK-based security labeling. Part of why we're moving more sockets to management by systemd is to centralize kernel-level mandatory access control (MAC) labeling implementations.

It means we have to support one implementation in systemd per MAC method rather than one implementation per MAC method, per daemon. Good systemd socket activation support for nginx means good support for multiple security toolkits.

[David Strauss]

Just noticed it, but the documentation page you linked says:
"The NGINX environment variable is used internally by nginx and should not be set directly by the user."

This basically seals the deal that Fedora, RHEL, Suse, Arch, and related distros aren't going to package socket activation support with nginx.

[Maxim Dounin]

What you are talking about is documentation and support policy. Right now NGINX environment variable (and the socket file descriptors it lists) is considered to be internal nginx mechanism, hence docs say what they say. If there is consensus that file descriptor passing is/may be usable by an external software like systemd - we might consider making public/documented (or introduce some other public/documented way to do the same).

[David Strauss]

Yes, that's exactly what I'm requesting: a supported, documented way to pass in file descriptors for sockets.

Using the NGINX environment variable directly with systemd is troublesome because a mismatch between the system socket and the "listen" directive causes nginx to ignore the systemd socket and try to open the socket itself. It ignores the systemd socket quietly because it sees the socket as an inherited one that's no longer in use in the current configuration. And, it's impossible to add better error reporting without distinguishing between sockets in the configuration that should always be inherited from ones that might be inherited because of internal hand-off during a reload.

My patches attempt to resolve this by providing a straightforward, non-redundant way to outsource a listener in nginx's configuration to an inherited systemd socket.

I've been keeping the discussion here despite mistakenly starting it here (rather than on the mailing list). Should I move it to raise this question of a supported means for public file descriptor inheritance?

Thank you for the guidance so far on where I should be proposing changes like this.

[Lennart Poettering]

I think it would be highly desirable on RHEL, Suse, and Fedora if nginx would support socket activation via systemd. As such I can only recommend supporting something like this. For cloud setups this allows spawning nginx instances on demand while overcommiting resources drastically.

The problem with the NGINX environment variable is that it is service-specific. We cannot really add support for that specific variable in systemd just for NGINX I fear. Note that many other 3rd party projects already understand systemd-style socket activation natively, and it would be great if nginx would be another one.

Note that the socket activation API we use for systemd is entirely generic, and non-systemd system managers can easily implement this too. it also compiles into NOPs on non-Linux systems, so that adding support for this will not break anything.

Hope this makes some sense...

[Anatol Pomozov]

Is there any chance to see this issue resolved?

I have following usecase. I have a pretty old hardware that I want to use as home security system storage/backup server. It also should serve a few small static HTML pages. I decided to install nginx to the server to serve HTML. Most of the time the server is unneeded, but only activated something like once a twice a week. I would prefer to use systemd great feature called socket activation. I want that nginx is started only on HTTP request. It will save some amount of resources. Yes, this amount is small but I am a control freak and want to run only bare minimum of processes needed on my server.

So it would be great to have socket activation for nginx. Are there any plans?

[est]

This may be offtopic but it would be awesome to support proxy_pass fd:4 too.

[sbaugh.twosigma.com@…]

For those of us that just care about passing a listening socket into Nginx, after digging into the source I saw that the NGINX environment variable takes a list of file descriptor separated by ":" or ";", with a trailing separator. (I think that used to be in the documentation but was removed at some point.)

So for my use case, passing the file descriptor as fd 3 and setting NGINX="3;" is working fine.

[tailhook@…]

Replying to sbaugh.twosigma.com@…:

For those of us that just care about passing a listening socket into Nginx, after digging into the source I saw that the NGINX environment variable takes a list of file descriptor separated by ":" or ";", with a trailing separator. (I think that used to be in the documentation but was removed at some point.)

So for my use case, passing the file descriptor as fd 3 and setting NGINX="3;" is working fine.

If you want this to work well, you need to set O_NONBLOCK on the socket, otherwise it will "mostly" work but create many issues with closing connections especially after configuration reload.

I'm not sure if setting O_NONBLOCK is supported by systemd (could not find in docs).

[tailhook@…]

Also, consider the following changes for this patch:

1. Instead of adding listen fd:3, use normal listen *:80 directive and check LISTEN_FDS
2. When LISTEN_FDS is set and LISTEN_PID equals to the current pid, enumerate all the sockets (3...3+LISTEN_FDS) and check matching addresses. Similarly to how NGINX=3;4; works now.

Here is why this is a good idea:

1. Config is the same with and without the socket activation
2. nginx -t isn't broken (usually command doesn't run under socket activation, so doesn't have fd:3 as a valid socket)
3. Works well with multiple listening addresses (not sure if systemd somehow keeps descriptors ordered)
4. Users will not mess up file descriptors just because of a mistake in a config file

[warpforge@…]

I'm not sure if setting O_NONBLOCK is supported by systemd (could not find in docs).

systemd configures the sockets as O_NONBLOCK if the corresponding service is configured to want non-blocking sockets [1]. This allows systemd to configure the socket appropriately even if the service handling the socket changes (e.g. Apache HTTPd vs. nginx handling port 80).

[1] ​https://www.freedesktop.org/software/systemd/man/systemd.service.html#NonBlocking=