nginx.pid is removed during reload if pid path is changed in nginx.conf but points to the same file through a symlink

[pdecat@…]

On recent debian systems and derivatives like Ubuntu, /var/run is a symlink to /run:

# ls -l /var/run                                                                                                                                                                                               
lrwxrwxrwx 1 root root 4 Jan 21  2015 /var/run -> /run

On such systems, if the path to the pid file is changed from /var/run/nginx.pid to /run/nginx.pid in nginx.conf, effectively pointing to the same file on the filesystem, then when the configuration is reloaded the new /run/nginx.pid file is created then destroyed when the old /var/run/nginx.pid is unlinked.

Following are the steps to reproduce this issue.

At first, the nginx service is running normally:

# grep nginx.pid /etc/nginx/nginx.conf 
pid /var/run/nginx.pid;
# cat /var/run/nginx.pid                                                                                                                                                                                       
1129
# ps -ef|grep nginx                                                                                                                                                                                            
root      1129     1  0 Sep21 ?        00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data  4290  1129  0 07:49 ?        00:00:00 nginx: worker process                           
www-data  4291  1129  0 07:49 ?        00:00:00 nginx: worker process                           
www-data  4292  1129  0 07:49 ?        00:00:00 nginx: worker process                           
www-data  4293  1129  0 07:49 ?        00:00:00 nginx: worker process                           

Simply reloading the configuration works fine:

# kill -HUP 1129                                                                                                                                                                                               
# ps -ef|grep nginx
root      1129     1  0 Sep21 ?        00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data  5171  1129  0 07:50 ?        00:00:00 nginx: worker process                           
www-data  5172  1129  0 07:50 ?        00:00:00 nginx: worker process                           
www-data  5173  1129  0 07:50 ?        00:00:00 nginx: worker process                           
www-data  5174  1129  0 07:50 ?        00:00:00 nginx: worker process                           
# cat /var/run/nginx.pid                                                                                                                                                                                       
1129

Changing the pid path from /var/run/nginx.pid to /run/nginx.pid then reloading the configuration exhibits the issue:

# sed -i 's/pid \/var\/run\/nginx.pid/pid \/run\/nginx.pid/' /etc/nginx/nginx.conf
# grep nginx.pid /etc/nginx/nginx.conf 
pid /run/nginx.pid;
# kill -HUP 1129
# ps -ef|grep nginx
root      1129     1  0 Sep21 ?        00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data  5950  1129  0 07:51 ?        00:00:00 nginx: worker process                           
www-data  5951  1129  0 07:51 ?        00:00:00 nginx: worker process                           
www-data  5952  1129  0 07:51 ?        00:00:00 nginx: worker process                           
www-data  5953  1129  0 07:51 ?        00:00:00 nginx: worker process                           
# cat /var/run/nginx.pid 
cat: /var/run/nginx.pid: No such file or directory

Using strace, this behaviour is explained because the previous path is unlinked after the new file is created:

    open("/var/run/nginx.pid", O_RDWR|O_CREAT|O_TRUNC, 0644) = 7
    pwrite(7, "1129\n", 5, 0)               = 5
    close(7)                                = 0
    unlink("/run/nginx.pid")                = 0

Issue first encountered on Debian Wheezy with nginx 1.6.2 from backports, then reproduced on Ubuntu with nginx 1.9.3.

[pdecat@…]

Oops, the above strace is from a subsequent test, the other way around (replacing /run/nginx.pid by /var/run/nginx.pid).

Here is the correct strace:

open("/run/nginx.pid", O_RDWR|O_CREAT|O_TRUNC, 0644) = 7
pwrite(7, "1129\n", 5, 0)               = 5
close(7)                                = 0
unlink("/var/run/nginx.pid")            = 0

[Maxim Dounin]

I don't think anything can be done with this: it's the file name that nginx checks to find out if the file is the same or not. If it's not the same, nginx removes previous pid file it created much like on normal exit, to ensure no stray files are left. Using a symlink confuses this test, and this results in the unlink() shown. As far as I understand, symlink also won't survive nginx restart, which makes it more or less useless.

If you think things can be improved - suggestions are welcome.

[Maxim Dounin]

Feedback timeout.

[pdecat@…]

Hi mdounin,

sorry I did not register to get email for changes regarding this issue,
so I did not receive the notice it was requiring feedback.

I guess the issue could be fixed by checking if the new and old filenames point to the same PID file, resolving symlink in the process and not deleting the PID file in that case.

Trying to restart nginx after the pid was destroyed will fail because the stopping part will not work, the old process will still be bound to the configured ports and the new process will fail to bind it.

[coreykasten@…]

I have posted code which fixes this issue to nginx-devel: ​http://mailman.nginx.org/pipermail/nginx-devel/2016-February/007961.html

I'm keenly interested in feedback.

[coreykasten@…]

Hi mdounin,

I never got a response on the dev mailing list regarding my posted patch. I'm curious if this will get fixed soon.

thanks!

[Maxim Dounin]

I don't think we are going to fix this. There are more than one way to screw things up, and trying to fix a particular one with complex logic does not seem to be the right way to go.