Some access log variables leak memory on HTTP/2 client connections
|Reported by:||Dylan Plecki||Owned by:|
|Keywords:||access_log memory leak||Cc:|
|uname -a:||Linux 6692e65482cc 4.9.184-linuxkit #1 SMP Tue Jul 2 22:58:16 UTC 2019 x86_64 GNU/Linux|
|nginx -V:||nginx version: nginx/1.17.4|
When used in the HTTP
log_format directive, variables such as
$server_addr cause a rapid increase of memory utilization in situations where a large number of requests are sent over a connection, most notably with HTTP/2. The memory usage seems to be tied to the client connection, since closing the connection brings the memory usage back down.
I believe this may be caused by
ngx_pnalloc calls within these variable methods, which allocate memory in the connection's memory pool, but do not free it until the connection is destroyed (see
src/http/ngx_http_variables.c:1364 as an example). This does not pass muster under HTTP/2, which may keep client connections open indefinitely while multiplexing many requests.
Reproduction files will be attached in a tar file. Please see the
README.txt file for repro steps.
This bug was found in an environment where nginx was used as a second-line proxy, behind another proxy that forwards all external requests over a small number of HTTP/2 connections to nginx. The memory leak was fast enough to exhaust 8GB of memory within 24 hours, while only processing a few hundred requests per second.