Nginx leaves UNIX domain sockets after SIGQUIT

[launchpad.net/~cpburnz]

According to the Nginx documentation, SIGQUIT will cause a "graceful shutdown" while SIGTERM will cause a "fast shutdown". If you send SIGQUIT to Nginx, it will leave behind stale UNIX domain socket files that were created using the listen directive. If there are any stale UNIX domain socket files when Nginx starts up, it will fail to listen on the socket because it already exists. However if you use SIGTERM, the UNIX domain socket files will be properly removed. I've encountered this with Nginx 1.6.2, 1.6.3, and 1.8.0 on Ubuntu 14.04.

Example /etc/nginx/nginx.conf:

http {
    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##

    gzip on;
    gzip_disable "msie6";

    ##
    # Virtual Host Configs
    ##

    include /etc/nginx/sites-enabled/*;
}

Example /etc/nginx/sites-enabled/serve-files:

server {
    listen unix:/run/serve-files.socket;
    root /var/www/files;
    location / {
        try_files $uri =404;
    }
}

Then start Nginx:

sudo nginx
# OR
sudo service nginx start

On first start, /run/serve-files.socket will be created because of the listen unix:/run/serve-files.socket; directive.

Then stop Nginx with SIGQUIT:

sudo kill -SIGQUIT $(cat /run/nginx.pid)
# OR
sudo service nginx stop # Sends SIGQUIT

The socket at /run/serve-files.socket will remain because it was not properly removed. If you try to restart Nginx, it will fail to start with the following logged to /var/log/nginx/error.log:

2015/04/24 10:16:27 [emerg] 5782#0: bind() to unix:/run/serve-files.socket failed (98: Address already in use)
2015/04/24 10:16:27 [emerg] 5782#0: bind() to unix:/run/serve-files.socket failed (98: Address already in use)
2015/04/24 10:16:27 [emerg] 5782#0: bind() to unix:/run/serve-files.socket failed (98: Address already in use)
2015/04/24 10:16:27 [emerg] 5782#0: bind() to unix:/run/serve-files.socket failed (98: Address already in use)
2015/04/24 10:16:27 [emerg] 5782#0: bind() to unix:/run/serve-files.socket failed (98: Address already in use)
2015/04/24 10:16:27 [emerg] 5782#0: still could not bind()

[Maxim Dounin]

The ngx_master_process_cycle() closes listening sockets by itself on SIGQUIT, without using the ngx_close_listening_sockets() function which is capable of correctly removing unix sockets after closing them. Looks like it needs to be changed to use ngx_close_listening_sockets() instead.

[openid.yandex.ru/yermulnik]

Encountered the same trouble with Nginx 1.9.1 on Ubuntu 14.04 and FreeBSD 10.1-RELEASE.
Please, consider fixing this ASAP.
Thanx.

[openid.yandex.ru/yermulnik]

Maxim Dounin, can you please let us know if this going to be implemented in a near future? thanx

[maxim]

Hello,

why it is so urgent?

[Maxim Dounin]

This problem should never affect any properly working nginx installation (there is no reason to stop nginx unless you are stopping the host itself), so it's unlikely to be looked into in a near future. Moreover, the fix is likely to affect other use cases (e.g., binary upgrade) and should be done with care, if at all.

[openid.yandex.ru/yermulnik]

So if for any reason I would decide to stop, deinstall and later install and start nginx without server reboot using some automation software like SlatStack, Chef etc, when I get broken node (or plenty of them in the case of mass reconfigurations) I should remember that the reason is "there is no reason to stop nginx unless you are stopping the host itself"? %-\
Anyway thanx for the answer.

[micah@…]

I can confirm that a SIGQUIT will not remove the socket, and a SIGTERM will.

However, I disagree with what mdounin says about this never affecting anyone. On a debian system, if you stop or restart nginx using the init script, you *do* run into this problem.

The initscript runs in its stop function:

start-stop-daemon --stop --quiet --retry=$STOP_SCHEDULE --pidfile $PID --name $NAME

The --stop argument will send a SIGTERM to all matching processes. However its possible to send a different signal with the --signal option, or the --retry option, and it seems like it is sending this option as $STOP_SCHEDULE which is by default set to 'QUIT/5' for systemd, and STOP_SCHEDULE="${STOP_SCHEDULE:-QUIT/5/TERM/5/KILL/5}" for the initscript.

in its restart function, it simply runs a stop() and then start().

However, a reload() will work ok for reloading the config.

This means that anyone doing a 'graceful shutdown' wont have their socket cleaned up. Most people would want to do a graceful shutdown before doing a 'fast' shutdown, no?

[micah@…]

It also should be noted that using SIGQUIT for process termination is a bit weird for process terminating:

SIGQUIT       3       Core    Quit from keyboard
SIGTERM      15       Term    Termination signal

most applications don't bother intercepting SIGQUIT.

[phaoost@…]

The defect still exists in version 1.14.1

[axos88@…]

Still exists on 1.16.0.

This is also preventing me from restarting nginx via systemctl.

[thibaultcha@…]

Ran into this issue myself. Here is a patch I just proposed to the nginx-devel mailing list that resolves it:

# HG changeset patch
# User Thibault Charbonnier <thibaultcha@me.com>
# Date 1582764433 28800
#      Wed Feb 26 16:47:13 2020 -0800
# Node ID 55ea1a9197a6f28d4da00909e5ea8585f6a08239
# Parent  4f18393a1d51bce6103ea2f1b2587900f349ba3d
Ensured SIGQUIT deletes listening UNIX socket files.

Prior to this patch, the SIGQUIT signal handling (graceful shutdown) did not
remove UNIX socket files since ngx_master_process_cycle reimplemented
listening
socket closings in lieu of using ngx_close_listening_sockets.

Since ngx_master_process_exit will call the aforementioned
ngx_close_listening_sockets, we can remove the custom implementation and now
expect listening sockets to be closed properly by
ngx_close_listening_sockets
instead.

This fixes the trac issue #753 (https://trac.nginx.org/nginx/ticket/753).

diff -r 4f18393a1d51 -r 55ea1a9197a6 src/os/unix/ngx_process_cycle.c
--- a/src/os/unix/ngx_process_cycle.c   Thu Feb 20 16:51:07 2020 +0300
+++ b/src/os/unix/ngx_process_cycle.c   Wed Feb 26 16:47:13 2020 -0800
@@ -77,12 +77,11 @@
     u_char            *p;
     size_t             size;
     ngx_int_t          i;
-    ngx_uint_t         n, sigio;
+    ngx_uint_t         sigio;
     sigset_t           set;
     struct itimerval   itv;
     ngx_uint_t         live;
     ngx_msec_t         delay;
-    ngx_listening_t   *ls;
     ngx_core_conf_t   *ccf;

     sigemptyset(&set);
@@ -205,16 +204,6 @@
             ngx_signal_worker_processes(cycle,

ngx_signal_value(NGX_SHUTDOWN_SIGNAL));

-            ls = cycle->listening.elts;
-            for (n = 0; n < cycle->listening.nelts; n++) {
-                if (ngx_close_socket(ls[n].fd) == -1) {
-                    ngx_log_error(NGX_LOG_EMERG, cycle->log,
ngx_socket_errno,
-                                  ngx_close_socket_n " %V failed",
-                                  &ls[n].addr_text);
-                }
-            }
-            cycle->listening.nelts = 0;
-
             continue;
         }

[Maxim Dounin]

Review can be seen here:

​http://mailman.nginx.org/pipermail/nginx-devel/2020-February/013017.html

Just for the record, here are links to the previous attempt to fix this and the review:

​http://mailman.nginx.org/pipermail/nginx-devel/2016-December/009207.html
​http://mailman.nginx.org/pipermail/nginx-devel/2016-December/009208.html
​http://mailman.nginx.org/pipermail/nginx-devel/2016-December/009239.html

[luisdavim@…]

This should be prioritised, it is an issue for example when running nginx in kubernetes.

​https://github.com/nginxinc/docker-nginx/issues/167
​https://github.com/nginxinc/docker-nginx/issues/377

[davehope@…]

Just a further comment to try and raise the profile of this issue. It's one I've been tracking for some time and face on a regular basis.

[Ruslan Ermilov <ru@…>]

In 7663:7cbf6389194b/nginx:

Fixed SIGQUIT not removing listening UNIX sockets (closes #753).

Listening UNIX sockets were not removed on graceful shutdown, preventing
the next runs. The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.