Opened 4 years ago

Closed 4 years ago

#1900 closed defect (invalid)

Nginx starting more worker processes when modsec rules are enabled

Reported by: marferiky@… Owned by:
Priority: major Milestone:
Component: nginx-core Version: 1.15.x
Keywords: modsec, worker processes Cc: marferiky@…
uname -a: 4.19.25-1.el7.centos.x86_64 #1 SMP Mon Feb 25 14:23:42 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.15.4
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC)
built with OpenSSL 1.0.2k-fips 26 Jan 2017
TLS SNI support enabled

Description

Hi there, we run Nginx in a Docker container as a sidecar in a pod in a Kubernetes cluster.

We have the ModSec module (version 3.0.3) enabled for Nginx.

In our Nginx configuration, we set the number of worker processes to be 16 (we have 16 cpu cores on our machines):

    worker_processes 16;
    events {
      worker_connections 1024;
    }

Without modsec enabled, it all works fine, we can see in the access logs that requests are processed fine, there are 16 worker processes running and the memory usage is not higher than 200-300MB.

However, when the modsec is enabled, the initial number of worker processes in nginx is 16, but over a period of 12 hours, the number of worker processes increases whenever a given modsec rule is matched by an HTTP request (in our case it is ruleID: 920350).

The number of worker processes increases until the memory limit (2GB, set at a kubernetes level) is met (at that point, nginx started up to 500 worker processes!)

When it hits the memory limit, nginx stops handling requests (according to the access log, we cannot see any requests being handled anymore). Furthermore, the issue is causing the entire machine where the nginx container is running to crash.

I am wondering why is nginx starting more worker processes than the configured number?

Attachments (1)

nginx_worker_processes.txt (22.1 KB ) - added by marferiky@… 4 years ago.
nginx worker processes

Download all attachments as: .zip

Change History (6)

comment:1 by maxim, 4 years ago

Hi,

can you check whether nginx demonstrates the same behaviour with modsecurity disabled?

I see nothing in OWASP ruleid 920350 here https://github.com/SpiderLabs/owasp-modsecurity-crs/ that can trigger such behaviour unless some subtle bug in the modsecurity module.

Thanks,

Maxim

by marferiky@…, 4 years ago

Attachment: nginx_worker_processes.txt added

nginx worker processes

comment:2 by marferiky@…, 4 years ago

hi Maxim, thanks for getting back!

When modsecurity is disabled, it all works fine: the number of worker processes in nginx is 16 as configured and the whole container memory usage is below 300MB.

The behavior is only happening when modsecurity is enabled and that rule matches HTTP requests

comment:3 by maxim, 4 years ago

I would update both nginx and, more important, libmodsecurity and nginx modsecurity module to the most recent snapshot from their v3/master branch and try to reproduce this issue again.

So far it sounds like a bug in the modsecurity and should be reported the modsecurity developers https://github.com/SpiderLabs/ModSecurity

Thanks,

Maxim

comment:4 by marferiky@…, 4 years ago

ok, I'll try that, thanks!

comment:5 by Maxim Dounin, 4 years ago

Resolution: invalid
Status: newclosed

Feedback timeout. Likely a bug in modsecurity.

Note: See TracTickets for help on using tickets.