Opened 3 weeks ago

Last modified 3 weeks ago

#2436 new defect

Nginx does log anything anymore

Reported by: jean-christophe-manciot@… Owned by:
Priority: major Milestone:
Component: nginx-core Version: 1.23.x
Keywords: log Cc:
uname -a: Linux hostname 5.19.0-30-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 6 15:40:20 UTC 2023 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.23.3 (x86_64-pc-linux-gnu)
built by gcc 12.2.0 (Debian 12.2.0-9)
built with OpenSSL 3.0.7 1 Nov 2022
TLS SNI support enabled
configure arguments: --build=x86_64-pc-linux-gnu --prefix=/usr --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/tmp/nginx/body --http-fastcgi-temp-path=/var/tmp/nginx/fastcgi --http-proxy-temp-path=/var/tmp/nginx/proxy --http-scgi-temp-path=/var/tmp/nginx/scgi --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi --lock-path=/run/lock/nginx.lock --modules-path=/usr/lib/nginx/modules --pid-path=/run/nginx.pid --sbin-path=/usr/sbin/nginx --group=www-data --user=www-data --add-dynamic-module=../git-ngx_headers_more --add-dynamic-module=../git-ngx_http_auth_pam_module --with-debug --with-file-aio --with-google_perftools_module --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_geoip_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_mp4_module --with-http_perl_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_xslt_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-perl_modules_path=/usr/lib/x86_64-linux-gnu/perl/5.36.0 --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_realip_module --with-stream_geoip_module --with-stream_ssl_preread_module --with-threads

Description

In /etc/nginx/nginx.conf:

user www-data;
http {
        log_format      main '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';
...
	include /etc/nginx/conf.d/*.conf;
	include /etc/nginx/sites-enabled/*;
}

In /etc/nginx/sites-enabled/site.conf:
server
{
...

access_log /var/log/nginx/site_access.log combined;
error_log /var/log/nginx/site_error.log warn;

}

In /var/log/nginx after accessing the site:

drwxr-xr-x 1 www-data adm              4096 Jan 13 10:11 ./
drwxrwxr-x 1 root     systemd-resolve 16384 Jan 13 10:11 ../
-rw-r--r-- 1 www-data root                0 Jan 13 10:11 access.log
-rw-r----- 1 www-data adm                 0 Jan 13 10:11 error.log
-rw-r----- 1 www-data adm                 0 Jan 13 10:11 site_access.log
-rw-r--r-- 1 www-data root                0 Jan 13 10:11 site_error.log

Sometimes, the access log is populated a little bit right after restarting nginx and accessing the site, but then the logging stops forever despite accessing the site.

To make sure there is no permissions issues:

# su -m www-data -c 'echo "testing" >> /var/log/nginx/site_error.log'
# cat /var/log/nginx/site_error.log
testing

If there is any link, in /etc/logrotate.d/nginx:

/var/log/nginx/*.log {
	daily
	missingok
	rotate 14
	compress
	delaycompress
	notifempty
	create 0640 www-data adm
	sharedscripts
	prerotate
		if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
			run-parts /etc/logrotate.d/httpd-prerotate; \
		fi \
	endscript
	postrotate
		invoke-rc.d nginx rotate >/dev/null 2>&1
	endscript
}

Also, the systemd service is running:

# systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; preset: enabled)
     Active: active (running) since Fri 2023-01-13 10:25:24 CET; 6min ago
       Docs: man:nginx(8)
    Process: 312868 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 312880 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Main PID: 312895 (nginx)
      Tasks: 5 (limit: 9226)
     Memory: 15.7M
        CPU: 2.010s
     CGroup: /system.slice/nginx.service
             ├─312895 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;"
             ├─312896 "nginx: worker process"
             ├─312897 "nginx: worker process"
             ├─312898 "nginx: worker process"
             └─312899 "nginx: worker process"

Jan 13 10:25:23 site systemd[1]: Starting A high performance web server and a reverse proxy server...
Jan 13 10:25:24 site systemd[1]: Started A high performance web server and a reverse proxy server.

And the nginx processes are owned by root and www-data:

# ps -elf | grep nginx
1 S root      375545       1  0  80   0 - 41675 sigsus 10:32 ?        00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
5 S www-data  375547  375545  0  80   0 - 42165 ep_pol 10:32 ?        00:00:00 nginx: worker process
5 S www-data  375549  375545  0  80   0 - 42056 ep_pol 10:32 ?        00:00:00 nginx: worker process
5 S www-data  375550  375545  0  80   0 - 42056 ep_pol 10:32 ?        00:00:00 nginx: worker process
5 S www-data  375551  375545  0  80   0 - 42089 ep_pol 10:32 ?        00:00:00 nginx: worker process

What is happening?

Change History (1)

comment:1 by Maxim Dounin, 3 weeks ago

What is happening?

First of all, I would consider making sure this isn't an issue with log rotation. For example, if logs are rotated, but nginx is not instructed to reopen logs, it will continue to use old log files for logging, which might look like nginx is not logging at all. In particular, note that invoke-rc.d nginx rotate >/dev/null 2>&1 in your logrotate configuration is likely wrong unless there is an actual init.d script for nginx.

Also, it might be a good idea to configure detailed error logging, such as debug at the global level, see here. This should help to distinguish between issues with logging (and log rotation) and issues with access log configuration in nginx.

The last but not least, ticket details indicate that you are using several 3rd party modules. Bugs in 3rd party modules can easily cause arbitrary unpredictable behaviour. Consider recompiling nginx without these modules to see if it helps. Alternatively, consider installing official package instead.

Note: See TracTickets for help on using tickets.