nginx-1.20.2 worker process memory usage keeps increasing

[rezoruno@…]

I have configured nginx-1.20.2 as a reverse proxy. The request traffic is max 200 req/s. Every time I check the memory usage of nginx: worker processes, usage keeps increasing. When I restart the nginx, memory usage back to normal, but it increasing by time.
I used nginx-1.20.1 for same manners for like 5 months with same configuration, but I did not come across a problem like this.
Any help will be appreciated. Thanks in advance.

[Maxim Dounin]

There are very few changes in 1.20.2 compared to 1.20.1, no idea what can cause increasing memory usage.

First of all, I would recommend to do the following to further debug this:

• Make sure by "memory usage" you actually mean nginx memory usage, as shown by "top" or "ps", and not a memory usage of the OS. Note that OS is likely to use any free memory to cache files on disk, and memory usage by OS is likely to keep increasing while nginx works as long as there are free memory.
• Make sure the memory usage increase you are observing isn't just a result of valid usage and does not stabilize over time. For example, in #2293 similar claim ("memory usage keep increasing") was diagnosed to be just a normal memory usage per configuration, which nginx was slowly reaching after a restart. Waiting for at least a couple of hours after a restart before checking memory usage is recommended.
• Check the libraries you use with nginx. If there are any changes, try to reproduce with the previously used libraries (or try nginx 1.20.1 with the updated libraries) to find out if the issue is caused by the library change.
• Make sure you observe the issue without any 3rd party modules.

It also might be a good idea to show your configuration to see if there are any potential issues.

Hope this helps.

[Maxim Dounin]

I've additionally review changes in nginx 1.20.2 and I think I see what can cause a memory leak. Not sure how to reproduce it though. Could you please test if the following patch fixes things for you?

diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c
--- a/src/http/ngx_http_request.c
+++ b/src/http/ngx_http_request.c
@@ -3751,6 +3751,8 @@ ngx_http_close_connection(ngx_connection
 #if (NGX_HTTP_SSL)
 
     if (c->ssl) {
+        c->ssl->shutdown_without_free = 0;
+
         if (ngx_ssl_shutdown(c) == NGX_AGAIN) {
             c->ssl->handler = ngx_http_close_connection;
             return;

[rezoruno@…]

I figured out the cause of memory leak. Actually, only difference between the old and new configuration was ssl_engine pkcs11 directive. Meanwhile, I am working on integrating nginx with HSM. In order to do so, this directive is enabled. When I disabled the directive, memory usage stabilizes. However, for integrating with HSM purpose I have to enable it. Addition to memory consuption problem I am facing with performance issues. At nginx error log, I am having SSL handshake errors espacially under the load. Example error log:
[crit] 58#58: *3313805 SSL_do_handshake() failed (SSL: error:8207A090:PKCS#11 module:pkcs11_private_encrypt:Operation active error:141EC044:SSL routines:tls_construct_server_key_exchange:internal error) while SSL handshaking, client: xx.xx.xx.xx, server: 0.0.0.0:443
What can lead to this, I know I wander off the topic but any suggestions/help would be greatly appreciated.

[Maxim Dounin]

Using HSM implies that private key operations are likely limited by the hardware module you are using, and performance issues aren't surprising. It is usually not recommended to use HSMs for anything under load.

As for memory issues and internal errors, these are to be looked in the OpenSSL engine being used. Upgrading to the most recent version of the PKCS11 engine might be the way to go.

One thing which might help as a workaround (though unlikely) is to avoid misusing ssl_engine to load the pkcs11 engine, and use appropriate OpenSSL configuration to initialize the engine instead (see #2271 or ​this test for examples, note init = 1).

[maxim]

Ticket retargeted after milestone closed

[Maxim Dounin]

Closing this. The issue as originally observed was reported to be cause by the pkcs11 engine. Also, additional investigation shows that the potential cause suspected in comment:2 cannot result in a memory leak.