Opened 4 years ago

Last modified 4 years ago

#796 reopened defect

nginx.pid is removed during reload if pid path is changed in nginx.conf but points to the same file through a symlink

Reported by: pdecat@… Owned by:
Priority: minor Milestone:
Component: nginx-core Version: 1.9.x
Keywords: Cc:
uname -a: Linux patrickxps 4.2.0-10-generic #12-Ubuntu SMP Tue Sep 15 19:43:01 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.9.3 (Ubuntu) built with OpenSSL 1.0.2d 9 Jul 2015 TLS SNI support enabled configure arguments: --with-cc-opt='-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_spdy_module --with-http_sub_module --with-http_xslt_module --with-mail --with-mail_ssl_module

Description

On recent debian systems and derivatives like Ubuntu, /var/run is a symlink to /run:

# ls -l /var/run                                                                                                                                                                                               
lrwxrwxrwx 1 root root 4 Jan 21  2015 /var/run -> /run

On such systems, if the path to the pid file is changed from /var/run/nginx.pid to /run/nginx.pid in nginx.conf, effectively pointing to the same file on the filesystem, then when the configuration is reloaded the new /run/nginx.pid file is created then destroyed when the old /var/run/nginx.pid is unlinked.

Following are the steps to reproduce this issue.

At first, the nginx service is running normally:

# grep nginx.pid /etc/nginx/nginx.conf 
pid /var/run/nginx.pid;
# cat /var/run/nginx.pid                                                                                                                                                                                       
1129
# ps -ef|grep nginx                                                                                                                                                                                            
root      1129     1  0 Sep21 ?        00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data  4290  1129  0 07:49 ?        00:00:00 nginx: worker process                           
www-data  4291  1129  0 07:49 ?        00:00:00 nginx: worker process                           
www-data  4292  1129  0 07:49 ?        00:00:00 nginx: worker process                           
www-data  4293  1129  0 07:49 ?        00:00:00 nginx: worker process                           

Simply reloading the configuration works fine:

# kill -HUP 1129                                                                                                                                                                                               
# ps -ef|grep nginx
root      1129     1  0 Sep21 ?        00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data  5171  1129  0 07:50 ?        00:00:00 nginx: worker process                           
www-data  5172  1129  0 07:50 ?        00:00:00 nginx: worker process                           
www-data  5173  1129  0 07:50 ?        00:00:00 nginx: worker process                           
www-data  5174  1129  0 07:50 ?        00:00:00 nginx: worker process                           
# cat /var/run/nginx.pid                                                                                                                                                                                       
1129

Changing the pid path from /var/run/nginx.pid to /run/nginx.pid then reloading the configuration exhibits the issue:

# sed -i 's/pid \/var\/run\/nginx.pid/pid \/run\/nginx.pid/' /etc/nginx/nginx.conf
# grep nginx.pid /etc/nginx/nginx.conf 
pid /run/nginx.pid;
# kill -HUP 1129
# ps -ef|grep nginx
root      1129     1  0 Sep21 ?        00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data  5950  1129  0 07:51 ?        00:00:00 nginx: worker process                           
www-data  5951  1129  0 07:51 ?        00:00:00 nginx: worker process                           
www-data  5952  1129  0 07:51 ?        00:00:00 nginx: worker process                           
www-data  5953  1129  0 07:51 ?        00:00:00 nginx: worker process                           
# cat /var/run/nginx.pid 
cat: /var/run/nginx.pid: No such file or directory

Using strace, this behaviour is explained because the previous path is unlinked after the new file is created:

    open("/var/run/nginx.pid", O_RDWR|O_CREAT|O_TRUNC, 0644) = 7
    pwrite(7, "1129\n", 5, 0)               = 5
    close(7)                                = 0
    unlink("/run/nginx.pid")                = 0

Issue first encountered on Debian Wheezy with nginx 1.6.2 from backports, then reproduced on Ubuntu with nginx 1.9.3.

Change History (7)

comment:1 Changed 4 years ago by pdecat@…

Oops, the above strace is from a subsequent test, the other way around (replacing /run/nginx.pid by /var/run/nginx.pid).

Here is the correct strace:

open("/run/nginx.pid", O_RDWR|O_CREAT|O_TRUNC, 0644) = 7
pwrite(7, "1129\n", 5, 0)               = 5
close(7)                                = 0
unlink("/var/run/nginx.pid")            = 0

comment:2 Changed 4 years ago by mdounin

I don't think anything can be done with this: it's the file name that nginx checks to find out if the file is the same or not. If it's not the same, nginx removes previous pid file it created much like on normal exit, to ensure no stray files are left. Using a symlink confuses this test, and this results in the unlink() shown. As far as I understand, symlink also won't survive nginx restart, which makes it more or less useless.

If you think things can be improved - suggestions are welcome.

comment:3 Changed 4 years ago by mdounin

  • Resolution set to wontfix
  • Status changed from new to closed

Feedback timeout.

comment:4 Changed 4 years ago by pdecat@…

Hi mdounin,

sorry I did not register to get email for changes regarding this issue,
so I did not receive the notice it was requiring feedback.

I guess the issue could be fixed by checking if the new and old filenames point to the same PID file, resolving symlink in the process and not deleting the PID file in that case.

Trying to restart nginx after the pid was destroyed will fail because the stopping part will not work, the old process will still be bound to the configured ports and the new process will fail to bind it.

comment:5 Changed 4 years ago by pdecat@…

  • Resolution wontfix deleted
  • Status changed from closed to reopened

comment:6 Changed 4 years ago by coreykasten@…

I have posted code which fixes this issue to nginx-devel: http://mailman.nginx.org/pipermail/nginx-devel/2016-February/007961.html

I'm keenly interested in feedback.

comment:7 Changed 4 years ago by coreykasten@…

Hi mdounin,

I never got a response on the dev mailing list regarding my posted patch. I'm curious if this will get fixed soon.

thanks!

Note: See TracTickets for help on using tickets.