Some access log variables leak memory on HTTP/2 client connections

[Dylan Plecki]

When used in the HTTP log_format directive, variables such as $server_port and $server_addr cause a rapid increase of memory utilization in situations where a large number of requests are sent over a connection, most notably with HTTP/2. The memory usage seems to be tied to the client connection, since closing the connection brings the memory usage back down.

I believe this may be caused by ngx_pnalloc calls within these variable methods, which allocate memory in the connection's memory pool, but do not free it until the connection is destroyed (see ngx_http_variable_server_port in src/http/ngx_http_variables.c:1364 as an example). This does not pass muster under HTTP/2, which may keep client connections open indefinitely while multiplexing many requests.

Reproduction files will be attached in a tar file. Please see the README.txt file for repro steps.

This bug was found in an environment where nginx was used as a second-line proxy, behind another proxy that forwards all external requests over a small number of HTTP/2 connections to nginx. The memory leak was fast enough to exhaust 8GB of memory within 24 hours, while only processing a few hundred requests per second.

[Maxim Dounin]

All connections in nginx are expected to be closed periodically, and this is done specifically to free all memory allocated from the connection's pool. In particular, HTTP/2 connections are closed per ​http2_max_requests, which is 1000 by default.

In the provided test the size difference between nginx worker processes using $server_port and $remote_port is (47360K - 35072K), that is, about 12 megabytes. This corresponds to 1000 connections, with 1000 requests in each, so it is maximum possible memory usage per connection, assuming default settings. And it is about 12 kilobytes per connection. While this might be noticeable in some tests, this looks negligible compared to other possible sources of memory usage in a typical HTTP/2 connection.

The "exhaust 8GB of memory within 24 hours" case you've observed in your environment most likely corresponds to http2_max_requests set to an arbitrary high value. Using arbitrary high values in http2_max_requests (as well as keepalive_requests) is indeed a bad idea, and could result in unexpected memory usage. Don't do that, it hurts. We may want to consider better documenting keepalive_requests and http2_max_requests to make sure people won't use arbitrary high values without thinking first.

Summing up the above, clearly there is no memory leak here. At most, it is slightly suboptimal memory usage in a particular configuration. And the serious effect observed is likely due to misconfiguration of http2_max_requests, which shouldn't be set to an arbitrary high value.