Opened 4 years ago

Closed 3 years ago

#1178 closed defect (fixed)

Inconsistent PID file paths cause systemctl timeout

Reported by: MrMYHuang@… Owned by: thresh
Priority: minor Milestone:
Component: nginx-package Version: 1.10.x
Keywords: Cc:
uname -a: Linux dsrl2.dsrl 3.10.0-327.36.1.el7.x86_64 #1 SMP Sun Sep 18 13:04:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.10.2
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-file-aio --with-threads --with-ipv6 --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_ssl_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic'

Description

My nginx is installed from nginx official yum repo. However, when I started nginx by the command:

systemctl start nginx

, it failed with timeout. See the error message:

systemctl status nginx
Jan 15 11:05:07 dsrl2.dsrl systemd[1]: PID file /run/nginx.pid not readable (yet?) after start.
Jan 15 11:07:06 dsrl2.dsrl systemd[1]: nginx.service start operation timed out. Terminating.
Jan 15 11:07:06 dsrl2.dsrl systemd[1]: Failed to start nginx - high performance web server.
Jan 15 11:07:06 dsrl2.dsrl systemd[1]: Unit nginx.service entered failed state.
Jan 15 11:07:06 dsrl2.dsrl systemd[1]: nginx.service failed.

I find the PID file path in /lib/systemd/system/nginx.service is different form /etc/nginx/nginx.conf. After I modify the path in nginx.conf, systemctl successfully starts nginx. Please fix this inconsistent PID file paths problem.

Change History (9)

comment:1 by Ilyas Bakirov, 4 years ago

Same problem, please see 1069, 884

Last edited 4 years ago by Ilyas Bakirov (previous) (diff)

in reply to:  1 comment:2 by MrMYHuang@…, 4 years ago

Replying to i.bakirov@…:

Same problem, please see 1069, 884

Ticket 1069 said the issue is harmless. But, it is harmful on my CentOS 7, because I find /var/run is not a symbolic link to /run in my CentOS 7. With the inconsistent PID paths, systemd will be timeout and then nginx process will be killed.
Additionally, I don't very understand the issue in 884 ...

Version 0, edited 4 years ago by MrMYHuang@… (next)

comment:3 by thresh, 4 years ago

because I find /var/run is not a symbolic link to /run in my CentOS 7

That's weird. Can you show the output of the following commands:

# rpm -q filesystem
# rpm -V filesystem
# mount | grep run
# ls -la /run/
# ls -la /var/run/

?

in reply to:  3 comment:4 by MrMYHuang@…, 4 years ago

Replying to thresh:

because I find /var/run is not a symbolic link to /run in my CentOS 7

That's weird. Can you show the output of the following commands:

# rpm -q filesystem
# rpm -V filesystem
# mount | grep run
# ls -la /run/
# ls -la /var/run/

?

Hi thresh,

This is the result:

[root@dsrl2 var]# rpm -q filesystem
filesystem-3.2-20.el7.x86_64
[root@dsrl2 var]# rpm -V filesystem
.....UG..    /usr/local/bin
.....UG..    /usr/local/share
.....UG..    /usr/local/share/applications
[root@dsrl2 var]# mount | grep run
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/user/42 type tmpfs (rw,nosuid,nodev,relatime,size=1631744k,mode=700,uid=42,gid=42)
gvfsd-fuse on /run/user/42/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=42,group_id=42)
tmpfs on /run/user/1035 type tmpfs (rw,nosuid,nodev,relatime,size=1631744k,mode=700,uid=1035,gid=1034)
gvfsd-fuse on /run/user/1035/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1035,group_id=1034)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=1631744k,mode=700,uid=1000,gid=1034)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1034)
tmpfs on /run/user/1034 type tmpfs (rw,nosuid,nodev,relatime,size=1631744k,mode=700,uid=1034,gid=1034)
tmpfs on /run/user/1044 type tmpfs (rw,nosuid,nodev,relatime,size=1631744k,mode=700,uid=1044,gid=1048)
gvfsd-fuse on /run/user/1044/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1044,group_id=1048)
tmpfs on /run/user/1042 type tmpfs (rw,nosuid,nodev,relatime,size=1631744k,mode=700,uid=1042,gid=1046)
gvfsd-fuse on /run/user/1042/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1042,group_id=1046)
tmpfs on /run/user/1041 type tmpfs (rw,nosuid,nodev,relatime,size=1631744k,mode=700,uid=1041,gid=1045)
gvfsd-fuse on /run/user/1041/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1041,group_id=1045)
gvfsd-fuse on /run/user/1034/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1034,group_id=1034)
[root@dsrl2 var]# ls -la /run/
total 16
drwxr-xr-x  21 root           root            560 Jan 15 11:35 .
dr-xr-xr-x. 21 root           root           4096 Dec 20 12:25 ..
drwxr-xr-x   3 root           lp               60 Dec 20 12:27 cups
prw-------   1 root           root              0 Dec 20 12:23 dmeventd-client
prw-------   1 root           root              0 Dec 20 12:23 dmeventd-server
drwx--x--x   4 root           gdm             120 Dec 20 12:27 gdm
srw-rw-rw-   1 root           root              0 Dec 20 12:27 gssproxy.sock
drwxr-xr-x   4 root           root            100 Dec 20 12:23 initramfs
drwxr-xr-x   6 root           root            120 Dec 20 12:27 lock
drwxr-xr-x   3 root           root             60 Dec 20 12:22 log
drwx------   2 root           root             80 Dec 20 12:23 lvm
-rw-r--r--   1 root           root              4 Dec 20 12:23 lvmetad.pid
drwxr-xr-x   2 root           root             60 Dec 20 12:22 mount
-rw-r--r--   1 root           root              5 Jan 15 11:35 nginx.pid
drwxr-xr-x   2 root           root             60 Dec 20 12:22 plymouth
drwx------   2 rpc            rpc              40 Dec 20 12:27 rpcbind
drwxr-xr-x   2 root           root             40 Dec 20 12:27 samba
drwx--x--x   2 setroubleshoot setroubleshoot   40 Dec 20 12:27 setroubleshoot
drwx------   2 root           root             40 Dec 20 12:27 svnserve
drwxr-xr-x   2 root           root             60 Dec 20 12:27 sysconfig
drwxr-xr-x  18 root           root            440 Jan 15 11:35 systemd
drwxr-xr-x   2 root           root             60 Dec 20 12:22 tmpfiles.d
drwxr-xr-x   2 root           root             60 Dec 20 12:27 tuned
drwxr-xr-x   7 root           root            160 Jan 15 11:40 udev
drwx------   2 root           root             40 Dec 20 12:27 udisks2
drwxr-xr-x   9 root           root            180 Jan 16 19:20 user
-rw-rw-r--   1 root           utmp              0 Dec 20 12:27 utmp
-rw-------   1 root           root              0 Dec 20 12:27 xtables.lock
[root@dsrl2 var]# ls -la /var/run/
total 184
drwxr-xr-x 24 root  root            4096 Jan 15 11:33 .
drwxr-xr-x 27 root  root            4096 Dec 20 12:23 ..
drwxr-xr-x  2 root  root            4096 Dec 20 12:27 abrt
-rw-------  1 root  root              11 Dec 19 13:01 alsactl.pid
-rw-r--r--  1 root  root               5 Dec 20 12:27 atd.pid
-rw-r--r--  1 root  root               4 Dec 20 12:27 auditd.pid
drwxr-xr-x  2 avahi avahi           4096 Aug 15  2015 avahi-daemon
drwxr-xr-x  2 root  root            4096 Nov 21  2015 certmonger
-rw-r--r--  1 root  root               4 Dec 20 12:27 chronyd.pid
-rw-r--r--  1 root  root               4 Feb 26  2015 chronyd.pid2
drwxr-xr-x  2 root  root            4096 Aug 18  2015 console
-rw-r--r--  1 root  root               5 Dec 20 12:27 crond.pid
----------  1 root  root               0 Feb 27  2015 cron.reboot
drwxr-xr-x  3 root  lp              4096 Dec 20 12:27 cups
drwxr-xr-x  2 root  root            4096 Dec 20 12:27 dbus
drwx------  4 root  root            4096 Aug 22 10:05 docker
-rw-r--r--  1 root  root               5 Aug 22 09:47 docker.pid
srw-rw----  1 root  docker             0 Aug 22 09:47 docker.sock
drwxr-xr-x  2 root  root            4096 Aug 18  2015 faillock
-rw-------  1 root  root               4 Dec 20 12:27 gssproxy.pid
drwxrwxr-x  3 root  libstoragemgmt  4096 Dec 20 12:27 lsm
drwx--x---  2 root  root            4096 Aug  3 00:28 mdadm
drwxrwxr-x  2 root  root            4096 Aug  3 00:15 netreport
drwxr-xr-x  2 root  root            4096 Dec 20 12:27 NetworkManager
drwx--x---  2 root  openvpn         4096 Dec 20 12:27 openvpn
drwxr-xr-x  2 root  root            4096 Nov 20  2015 plymouth
drwxr-xr-x  4 root  root            4096 Feb 27  2015 pm-utils
drwxr-xr-x  2 root  root            4096 Feb 27  2015 ppp
-r--r--r--  1 root  root               0 Feb 27  2015 rpcbind.lock
srw-rw-rw-  1 root  root               0 Dec 20 12:27 rpcbind.sock
-rw-r--r--  1 root  root               0 Jan 29  2016 run-texhash
drwxr-xr-x  2 root  root            4096 Feb 27  2015 samba
srw-rw-rw-  1 root  root               0 Nov 17  2015 sdp
drwxr-xr-x  2 root  root            4096 Aug 18  2015 sepermit
drwxr-xr-x  2 root  root            4096 Feb 27  2015 setrans
-rw-------  1 root  root               5 Feb 27  2015 sm-notify.pid
drwxr-xr-x  2 root  root            4096 Nov 20  2015 spice-vdagentd
-rw-r--r--  1 root  root               5 Dec 20 12:27 sshd.pid
-rw-------  1 root  root               4 Dec 20 12:27 syslogd.pid
drwxr-xr-x  2 root  root            4096 May  5  2016 texlive
-rw-rw-r--  1 root  utmp           35328 Jan 16 15:06 utmp
drwxr-xr-x  2 root  root            4096 Jun 16  2015 wpa_supplicant
-rw-r--r--  1 root  root               5 Dec 20 12:27 ypbind.pid
-rw-r--r--  1 root  root               5 Dec 20 12:27 ypserv.pid

comment:5 by thresh, 4 years ago

Interesting. It seems something went really wrong when installing your OS, because filesystem package creates /run and symlinks /var/run into it:

https://git.centos.org/blob/rpms!filesystem/d16f1df2bacd5f61fc7495c3d0af3c5ecea43ccf/SPECS!filesystem.spec#L48

I would guess with such a broken install you will encounter many more issues than just nginx, so I'd advise you to fix your OS, instead of introducing /var/run/nginx.pid to nginx.service (which contradicts current CentOS/RHEL guidelines: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Migration_Planning_Guide/sect-Red_Hat_Enterprise_Linux-Migration_Planning_Guide-File_System_Layout.html)

in reply to:  5 comment:6 by MrMYHuang@…, 4 years ago

Replying to thresh:

Interesting. It seems something went really wrong when installing your OS, because filesystem package creates /run and symlinks /var/run into it:

https://git.centos.org/blob/rpms!filesystem/d16f1df2bacd5f61fc7495c3d0af3c5ecea43ccf/SPECS!filesystem.spec#L48

Thanks for your information. I think of that I didn't install CentOS by ISO on my this PC, but by a tar of another PC, which installed CentOS by ISO.

I would guess with such a broken install you will encounter many more issues than just nginx, so I'd advise you to fix your OS, instead of introducing /var/run/nginx.pid to nginx.service (which contradicts current CentOS/RHEL guidelines: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Migration_Planning_Guide/sect-Red_Hat_Enterprise_Linux-Migration_Planning_Guide-File_System_Layout.html)

Actually, this is the first time I encounter the /var/run symlink issue...
I have no answer whether the inconsistent paths of the nginx rpm should be made consistent or not. Nevertheless, my nginx now works.

comment:7 by Maxim Dounin, 3 years ago

Component: othernginx-package

comment:8 by thresh, 3 years ago

Owner: set to thresh
Status: newassigned

I've made the paths consistent in the packaging sources; will be available on the next nginx releases (1.12.1 and 1.13.2).

comment:9 by Sergey Budnevitch, 3 years ago

Resolution: fixed
Status: assignedclosed
Note: See TracTickets for help on using tickets.