Opened 5 weeks ago

Closed 3 weeks ago

Last modified 3 weeks ago

#1873 closed defect (invalid)

possible memory leak in stream modules

Reported by: honghaier250@… Owned by:
Priority: critical Milestone:
Component: nginx-module Version: 1.13.x
Keywords: memory leak in stream modules Cc: gcc, version, 7.4.0, (Ubuntu, 7.4.0-1ubuntu1~18.04.1)
uname -a: Linux ubuntu 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.13.6.2 built by gcc 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1) built with OpenSSL 1.0.2o 27 Mar 2018 (running with OpenSSL 1.0.2o 27 Mar 2018 []) TLS SNI support enabled configure arguments: --with-debug --with-cc-opt='-DNGX_LUA_USE_ASSERT -DNGX_LUA_ABORT_AT_PANIC -O2 -g3 -O0 -Wall -Werror -I/usr/local/ssl/include' --with-ld-opt='-Wl,-rpath,/kssl/TRP/luajit/lib -L/usr/local/ssl/lib -lssl -lcrypto' --with-http_sub_module --with-stream_ssl_preread_module --with-ipv6 --http-fastcgi-temp-path=/dev/null --http-uwsgi-temp-path=/dev/null --http-scgi-temp-path=/dev/null --http-client-body-temp-path=logs/client-body-temp --http-proxy-temp-path=logs/proxy-temp --with-stream --with-stream_ssl_module --with-http_ssl_module

Description

Hello everyone, I found that the nginx( version 1.13.2 ~ 1.17.x) stream module may have a memory leak problem, as follows:

Memory usage of the worker process when the service starts

top - 00:15:50 up 7 days, 14:39, 13 users,  load average: 0.11, 0.07, 0.02
任务:   2 total,   0 running,   2 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.5 us,  2.0 sy,  0.0 ni, 93.3 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem : 16316024 total,  2307824 free,  5340828 used,  8667372 buff/cache
KiB Swap:  1003516 total,   866044 free,   137472 used.  9180580 avail Mem

进USER      PR  NI    VIRT    RES    SHR  %CPU %MEM     TIME+ COMMAND
 9483 root      20   0  295016 175632   4580 S   0.0  1.1   0:00.07 nginx

pmap statistics when the service starts

9483:   nginx: worker process                                                               
          住址 Perm   Offset Device    Inode   Size    Rss    Pss Referenced Anonymous LazyFree ShmemPmdMapped Shared_Hugetlb Private_Hugetlb Swap SwapPss Locked Mapping
        4034c000 rw-p 00000000  00:00        0    128     60     30          0        60        0              0              0               0    0       0      0 
    556dc6e93000 r-xp 00000000  fd:00  2621849   3048    768    640        768         0        0              0              0               0    0       0      0 /kssl/TRP/nginx/sbin/nginx
    556dc738c000 r--p 002f9000  fd:00  2621849      8      8      4          8         8        0              0              0               0    0       0      0 /kssl/TRP/nginx/sbin/nginx
    556dc738e000 rw-p 002fb000  fd:00  2621849    128    128     68        108       128        0              0              0               0    0       0      0 /kssl/TRP/nginx/sbin/nginx
    556dc73ae000 rw-p 00000000  00:00        0    144     24     20         16        24        0              0              0               0    0       0      0 
    556dc7a5b000 rw-p 00000000  00:00        0    536    432    260        104       432        0              0              0               0    0       0      0 [heap]
    7f5acfad8000 rw-p 00000000  00:00        0 169612 169600 169600     169600    169600        0              0              0               0    0       0      0 
    7f5ada07b000 r-xp 00000000  fd:00  5509942     28     28      0         28         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/librt-2.27.so
    7f5ada082000 ---p 00007000  fd:00  5509942   2044      0      0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/librt-2.27.so
    7f5ada281000 r--p 00006000  fd:00  5509942      4      4      4          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/librt-2.27.so
    7f5ada282000 rw-p 00007000  fd:00  5509942      4      4      4          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/librt-2.27.so
    7f5ada283000 r-xp 00000000  fd:00  5505255    240     64     12         64         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_systemd.so.2
    7f5ada2bf000 ---p 0003c000  fd:00  5505255   2044      0      0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_systemd.so.2
    7f5ada4be000 r--p 0003b000  fd:00  5505255     12     12     12         12        12        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_systemd.so.2
    7f5ada4c1000 rw-p 0003e000  fd:00  5505255      4      4      4          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_systemd.so.2
    7f5ada4c2000 rw-s 00000000  00:05 22171283  16384      0      0          0         0        0              0              0               0    0       0      0 /dev/zero (deleted)
    7f5adb4c2000 r-xp 00000000  fd:00  5509891     44     44      0         44         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_files-2.27.so
    7f5adb4cd000 ---p 0000b000  fd:00  5509891   2044      0      0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_files-2.27.so
    7f5adb6cc000 r--p 0000a000  fd:00  5509891      4      4      2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_files-2.27.so
    7f5adb6cd000 rw-p 0000b000  fd:00  5509891      4      4      2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_files-2.27.so
    7f5adb6ce000 rw-p 00000000  00:00        0     24      0      0          0         0        0              0              0               0    0       0      0 
    7f5adb6d4000 r-xp 00000000  fd:00  5509885     92     64      1         64         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnsl-2.27.so
    7f5adb6eb000 ---p 00017000  fd:00  5509885   2044      0      0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnsl-2.27.so
    7f5adb8ea000 r--p 00016000  fd:00  5509885      4      4      2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnsl-2.27.so
    7f5adb8eb000 rw-p 00017000  fd:00  5509885      4      4      2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnsl-2.27.so
    7f5adb8ec000 rw-p 00000000  00:00        0      8      0      0          0         0        0              0              0               0    0       0      0 
    7f5adb8ee000 r-xp 00000000  fd:00  5509902     44     44      1         44         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_nis-2.27.so
    7f5adb8f9000 ---p 0000b000  fd:00  5509902   2044      0      0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_nis-2.27.so
    7f5adbaf8000 r--p 0000a000  fd:00  5509902      4      4      2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_nis-2.27.so
    7f5adbaf9000 rw-p 0000b000  fd:00  5509902      4      4      2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_nis-2.27.so
    7f5adbafa000 r-xp 00000000  fd:00  5509887     32     32      0         32         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_compat-2.27.so
    7f5adbb02000 ---p 00008000  fd:00  5509887   2048      0      0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_compat-2.27.so
    7f5adbd02000 r--p 00008000  fd:00  5509887      4      4      2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_compat-2.27.so
    7f5adbd03000 rw-p 00009000  fd:00  5509887      4      4      4          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_compat-2.27.so
    7f5adbd04000 r-xp 00000000  fd:00  8396806     28     28      0         28         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
    7f5adbd0b000 ---p 00007000  fd:00  8396806   2044      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
    7f5adbf0a000 r--p 00006000  fd:00  8396806      4      4      2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
    7f5adbf0b000 rw-p 00007000  fd:00  8396806      4      4      2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
    7f5adbf0c000 r-xp 00000000  fd:00  8397004   1040     80      4         80         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
    7f5adc010000 ---p 00104000  fd:00  8397004   2044      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
    7f5adc20f000 r--p 00103000  fd:00  8397004     12     12      6          4        12        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
    7f5adc212000 rw-p 00106000  fd:00  8397004      8      8      4          0         8        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
    7f5adc214000 rw-p 00000000  00:00        0      4      0      0          0         0        0              0              0               0    0       0      0 
    7f5adc215000 r-xp 00000000  fd:00  8397052    280     64      4         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
    7f5adc25b000 ---p 00046000  fd:00  8397052   2044      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
    7f5adc45a000 r--p 00045000  fd:00  8397052     12     12      6          4        12        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
    7f5adc45d000 rw-p 00048000  fd:00  8397052      4      4      2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
    7f5adc45e000 rw-p 00000000  00:00        0      4      0      0          0         0        0              0              0               0    0       0      0 
    7f5adc45f000 r-xp 00000000  fd:00  8397036     56     56      4         56         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
    7f5adc46d000 ---p 0000e000  fd:00  8397036   2044      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
    7f5adc66c000 r--p 0000d000  fd:00  8397036      4      4      2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
    7f5adc66d000 rw-p 0000e000  fd:00  8397036      4      4      2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
    7f5adc66e000 r-xp 00000000  fd:00  8397612    160     12      0         12         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
    7f5adc696000 ---p 00028000  fd:00  8397612   2044      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
    7f5adc895000 r--p 00027000  fd:00  8397612      4      4      2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
    7f5adc896000 rw-p 00028000  fd:00  8397612      4      4      2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
    7f5adc897000 r-xp 00000000  fd:00  8396906    508     64      2         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.2
    7f5adc916000 ---p 0007f000  fd:00  8396906   2048      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.2
    7f5adcb16000 r--p 0007f000  fd:00  8396906      4      4      2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.2
    7f5adcb17000 rw-p 00080000  fd:00  8396906      4      4      2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.2
    7f5adcb18000 r-xp 00000000  fd:00  8397040    204     64      2         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.4
    7f5adcb4b000 ---p 00033000  fd:00  8397040   2044      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.4
    7f5adcd4a000 r--p 00032000  fd:00  8397040      4      4      2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.4
    7f5adcd4b000 rw-p 00033000  fd:00  8397040      4      4      2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.4
    7f5adcd4c000 r-xp 00000000  fd:00  8397242    208     76      7         76         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libnettle.so.6.4
    7f5adcd80000 ---p 00034000  fd:00  8397242   2044      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libnettle.so.6.4
    7f5adcf7f000 r--p 00033000  fd:00  8397242      8      8      4          4         8        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libnettle.so.6.4
    7f5adcf81000 rw-p 00035000  fd:00  8397242      4      4      2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libnettle.so.6.4
    7f5adcf82000 r-xp 00000000  fd:00  8397496     68     64      3         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.5
    7f5adcf93000 ---p 00011000  fd:00  8397496   2048      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.5
    7f5add193000 r--p 00011000  fd:00  8397496      4      4      2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.5
    7f5add194000 rw-p 00012000  fd:00  8397496      4      4      2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.5
    7f5add195000 r-xp 00000000  fd:00  8397166   1512     64      2         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libunistring.so.2.1.0
    7f5add30f000 ---p 0017a000  fd:00  8397166   2048      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libunistring.so.2.1.0
    7f5add50f000 r--p 0017a000  fd:00  8397166     12     12      6          4        12        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libunistring.so.2.1.0
    7f5add512000 rw-p 0017d000  fd:00  8397166      4      4      2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libunistring.so.2.1.0
    7f5add513000 r-xp 00000000  fd:00  8397082    112     64      3         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.3
    7f5add52f000 ---p 0001c000  fd:00  8397082   2044      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.3
    7f5add72e000 r--p 0001b000  fd:00  8397082      4      4      2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.3
    7f5add72f000 rw-p 0001c000  fd:00  8397082      4      4      2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.3
    7f5add730000 r-xp 00000000  fd:00  8397280   1128    128      8        128         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
    7f5add84a000 ---p 0011a000  fd:00  8397280   2048      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
    7f5adda4a000 r--p 0011a000  fd:00  8397280     40     40     20          4        40        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
    7f5adda54000 rw-p 00124000  fd:00  8397280     40     40     22          4        40        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
    7f5adda5e000 rw-p 00000000  00:00        0      4      4      2          0         4        0              0              0               0    0       0      0 
    7f5adda5f000 r-xp 00000000  fd:00  8397399     84     68      9         68         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
    7f5adda74000 ---p 00015000  fd:00  8397399   2044      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
    7f5addc73000 r--p 00014000  fd:00  8397399      4      4      2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
    7f5addc74000 rw-p 00015000  fd:00  8397399      4      4      2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
    7f5addc75000 r-xp 00000000  fd:00  8397034    204     64      4         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
    7f5addca8000 ---p 00033000  fd:00  8397034   2044      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
    7f5addea7000 r--p 00032000  fd:00  8397034      8      8      4          4         8        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
    7f5addea9000 rw-p 00034000  fd:00  8397034      4      4      2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
    7f5addeaa000 rw-p 00000000  00:00        0      4      0      0          0         0        0              0              0               0    0       0      0 
    7f5addeab000 r-xp 00000000  fd:00  5505070     12     12      1         12         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libcom_err.so.2.1
    7f5addeae000 ---p 00003000  fd:00  5505070   2044      0      0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libcom_err.so.2.1
    7f5ade0ad000 r--p 00002000  fd:00  5505070      4      4      2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libcom_err.so.2.1
    7f5ade0ae000 rw-p 00003000  fd:00  5505070      4      4      2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libcom_err.so.2.1
    7f5ade0af000 r-xp 00000000  fd:00  8396540    632    124      9        124         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
    7f5ade14d000 ---p 0009e000  fd:00  8396540   2048      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
    7f5ade34d000 r--p 0009e000  fd:00  8396540      4      4      2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
    7f5ade34e000 rw-p 0009f000  fd:00  8396540     12     12      6          0        12        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
    7f5ade351000 r-xp 00000000  fd:00  8397137    540    124      9        124         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
    7f5ade3d8000 ---p 00087000  fd:00  8397137   2044      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
    7f5ade5d7000 r--p 00086000  fd:00  8397137     16     16      8          4        16        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
    7f5ade5db000 rw-p 0008a000  fd:00  8397137      8      8      4          0         8        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
    7f5ade5dd000 rw-p 00000000  00:00        0      4      0      0          0         0        0              0              0               0    0       0      0 
    7f5ade5de000 r-xp 00000000  fd:00  8397038     32     32      2         32         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
    7f5ade5e6000 ---p 00008000  fd:00  8397038   2044      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
    7f5ade7e5000 r--p 00007000  fd:00  8397038      4      4      2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
    7f5ade7e6000 rw-p 00008000  fd:00  8397038      4      4      2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
    7f5ade7e7000 r-xp 00000000  fd:00  8405969   1376    164      8        164         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.14.10
    7f5ade93f000 ---p 00158000  fd:00  8405969   2044      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.14.10
    7f5adeb3e000 r--p 00157000  fd:00  8405969     48     48     24          4        48        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.14.10
    7f5adeb4a000 rw-p 00163000  fd:00  8405969      4      4      4          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.14.10
    7f5adeb4b000 rw-p 00000000  00:00        0      4      4      2          0         4        0              0              0               0    0       0      0 
    7f5adeb4c000 r-xp 00000000  fd:00  8396961    244     64      4         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
    7f5adeb89000 ---p 0003d000  fd:00  8396961   2048      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
    7f5aded89000 r--p 0003d000  fd:00  8396961      8      8      4          4         8        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
    7f5aded8b000 rw-p 0003f000  fd:00  8396961      8      8      4          0         8        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
    7f5aded8d000 r-xp 00000000  fd:00  8397423    100     64      6         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
    7f5adeda6000 ---p 00019000  fd:00  8397423   2048      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
    7f5adefa6000 r--p 00019000  fd:00  8397423      4      4      2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
    7f5adefa7000 rw-p 0001a000  fd:00  8397423      4      4      2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
    7f5adefa8000 r-xp 00000000  fd:00  5509940     92     60      0         60         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libresolv-2.27.so
    7f5adefbf000 ---p 00017000  fd:00  5509940   2048      0      0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libresolv-2.27.so
    7f5adf1bf000 r--p 00017000  fd:00  5509940      4      4      2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libresolv-2.27.so
    7f5adf1c0000 rw-p 00018000  fd:00  5509940      4      4      2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libresolv-2.27.so
    7f5adf1c1000 rw-p 00000000  00:00        0      8      0      0          0         0        0              0              0               0    0       0      0 
    7f5adf1c3000 r-xp 00000000  fd:00  8389082     52     52      3         52         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.8
    7f5adf1d0000 ---p 0000d000  fd:00  8389082   2044      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.8
    7f5adf3cf000 r--p 0000c000  fd:00  8389082      4      4      2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.8
    7f5adf3d0000 rw-p 0000d000  fd:00  8389082      4      4      2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.8
    7f5adf3d1000 r-xp 00000000  fd:00  5505267     92     64      3         64         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f5adf3e8000 ---p 00017000  fd:00  5505267   2044      0      0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f5adf5e7000 r--p 00016000  fd:00  5505267      4      4      2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f5adf5e8000 rw-p 00017000  fd:00  5505267      4      4      2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f5adf5e9000 r-xp 00000000  fd:00  8389083    312     64      4         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8
    7f5adf637000 ---p 0004e000  fd:00  8389083   2044      0      0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8
    7f5adf836000 r--p 0004d000  fd:00  8389083      8      8      4          4         8        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8
    7f5adf838000 rw-p 0004f000  fd:00  8389083      4      4      2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8
    7f5adf839000 rw-p 00000000  00:00        0      8      0      0          0         0        0              0              0               0    0       0      0 
    7f5adf83b000 r-xp 00000000  fd:00  5509801   1948   1424      8       1424         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libc-2.27.so
    7f5adfa22000 ---p 001e7000  fd:00  5509801   2048      0      0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libc-2.27.so
    7f5adfc22000 r--p 001e7000  fd:00  5509801     16     16      8          8        16        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libc-2.27.so
    7f5adfc26000 rw-p 001eb000  fd:00  5509801      8      8      8          8         8        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libc-2.27.so
    7f5adfc28000 rw-p 00000000  00:00        0     16     16     16         16        16        0              0              0               0    0       0      0 
    7f5adfc2c000 r-xp 00000000  fd:00  5509973    112     64      0         64         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libz.so.1.2.11
    7f5adfc48000 ---p 0001c000  fd:00  5509973   2044      0      0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libz.so.1.2.11
    7f5adfe47000 r--p 0001b000  fd:00  5509973      4      4      2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libz.so.1.2.11
    7f5adfe48000 rw-p 0001c000  fd:00  5509973      4      4      2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libz.so.1.2.11
    7f5adfe49000 r-xp 00000000  fd:00  5509923    448     64      0         64         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libpcre.so.3.13.3
    7f5adfeb9000 ---p 00070000  fd:00  5509923   2048      0      0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libpcre.so.3.13.3
    7f5ae00b9000 r--p 00070000  fd:00  5509923      4      4      2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libpcre.so.3.13.3
    7f5ae00ba000 rw-p 00071000  fd:00  5509923      4      4      2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libpcre.so.3.13.3
    7f5ae00bb000 r-xp 00000000  fd:00  5509864   1652     84      1         84         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libm-2.27.so
    7f5ae0258000 ---p 0019d000  fd:00  5509864   2044      0      0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libm-2.27.so
    7f5ae0457000 r--p 0019c000  fd:00  5509864      4      4      2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libm-2.27.so
    7f5ae0458000 rw-p 0019d000  fd:00  5509864      4      4      2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libm-2.27.so
    7f5ae0459000 r-xp 00000000  fd:00  2621865    620     64     64         64         0        0              0              0               0    0       0      0 /kssl/TRP/luajit/lib/libluajit-5.1.so.2.1.0
    7f5ae04f4000 ---p 0009b000  fd:00  2621865   2044      0      0          0         0        0              0              0               0    0       0      0 /kssl/TRP/luajit/lib/libluajit-5.1.so.2.1.0
    7f5ae06f3000 r--p 0009a000  fd:00  2621865      8      8      4          4         8        0              0              0               0    0       0      0 /kssl/TRP/luajit/lib/libluajit-5.1.so.2.1.0
    7f5ae06f5000 rw-p 0009c000  fd:00  2621865      4      4      2          0         4        0              0              0               0    0       0      0 /kssl/TRP/luajit/lib/libluajit-5.1.so.2.1.0
    7f5ae06f6000 r-xp 00000000  fd:00  5509811     36     36      1         36         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libcrypt-2.27.so
    7f5ae06ff000 ---p 00009000  fd:00  5509811   2044      0      0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libcrypt-2.27.so
    7f5ae08fe000 r--p 00008000  fd:00  5509811      4      4      2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libcrypt-2.27.so
    7f5ae08ff000 rw-p 00009000  fd:00  5509811      4      4      2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libcrypt-2.27.so
    7f5ae0900000 rw-p 00000000  00:00        0    184      0      0          0         0        0              0              0               0    0       0      0 
    7f5ae092e000 r-xp 00000000  fd:00  5509934    104     72      0         72         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libpthread-2.27.so
    7f5ae0948000 ---p 0001a000  fd:00  5509934   2044      0      0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libpthread-2.27.so
    7f5ae0b47000 r--p 00019000  fd:00  5509934      4      4      2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libpthread-2.27.so
    7f5ae0b48000 rw-p 0001a000  fd:00  5509934      4      4      4          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libpthread-2.27.so
    7f5ae0b49000 rw-p 00000000  00:00        0     16      4      4          4         4        0              0              0               0    0       0      0 
    7f5ae0b4d000 r-xp 00000000  fd:00  5509824     12     12      0         12         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libdl-2.27.so
    7f5ae0b50000 ---p 00003000  fd:00  5509824   2044      0      0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libdl-2.27.so
    7f5ae0d4f000 r--p 00002000  fd:00  5509824      4      4      2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libdl-2.27.so
    7f5ae0d50000 rw-p 00003000  fd:00  5509824      4      4      2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libdl-2.27.so
    7f5ae0d51000 r-xp 00000000  fd:00  9309093     32     28     28         28         0        0              0              0               0    0       0      0 /usr/local/ssl/lib/libKLCRL.so.1.0.0
    7f5ae0d59000 ---p 00008000  fd:00  9309093   2044      0      0          0         0        0              0              0               0    0       0      0 /usr/local/ssl/lib/libKLCRL.so.1.0.0
    7f5ae0f58000 r--p 00007000  fd:00  9309093      4      4      2          4         4        0              0              0               0    0       0      0 /usr/local/ssl/lib/libKLCRL.so.1.0.0
    7f5ae0f59000 rw-p 00008000  fd:00  9309093      4      4      2          0         4        0              0              0               0    0       0      0 /usr/local/ssl/lib/libKLCRL.so.1.0.0
    7f5ae0f5a000 r-xp 00000000  fd:00  9309081   2564    128     16        128         0        0              0              0               0    0       0      0 /usr/local/ssl/lib/libcrypto.so.1.0.0
    7f5ae11db000 ---p 00281000  fd:00  9309081   2044      0      0          0         0        0              0              0               0    0       0      0 /usr/local/ssl/lib/libcrypto.so.1.0.0
    7f5ae13da000 r--p 00280000  fd:00  9309081    152    140     70          4       140        0              0              0               0    0       0      0 /usr/local/ssl/lib/libcrypto.so.1.0.0
    7f5ae1400000 rw-p 002a6000  fd:00  9309081     48     48     24          0        48        0              0              0               0    0       0      0 /usr/local/ssl/lib/libcrypto.so.1.0.0
    7f5ae140c000 rw-p 00000000  00:00        0     16     16      8          0        16        0              0              0               0    0       0      0 
    7f5ae1410000 r-xp 00000000  fd:00  9309084    560     64     64         64         0        0              0              0               0    0       0      0 /usr/local/ssl/lib/libssl.so.1.0.0
    7f5ae149c000 ---p 0008c000  fd:00  9309084   2048      0      0          0         0        0              0              0               0    0       0      0 /usr/local/ssl/lib/libssl.so.1.0.0
    7f5ae169c000 r--p 0008c000  fd:00  9309084     16     16      8          4        16        0              0              0               0    0       0      0 /usr/local/ssl/lib/libssl.so.1.0.0
    7f5ae16a0000 rw-p 00090000  fd:00  9309084     28     28     14          0        28        0              0              0               0    0       0      0 /usr/local/ssl/lib/libssl.so.1.0.0
    7f5ae16a7000 rw-p 00000000  00:00        0      4      4      2          0         4        0              0              0               0    0       0      0 
    7f5ae16a8000 r-xp 00000000  fd:00  5509773    156    156      0        156         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/ld-2.27.so
    7f5ae189b000 rw-p 00000000  00:00        0     68     64     38         56        64        0              0              0               0    0       0      0 
    7f5ae18cc000 rw-s 00000000  00:05 22171285      4      0      0          0         0        0              0              0               0    0       0      0 /dev/zero (deleted)
    7f5ae18cd000 rw-p 00000000  00:00        0      8      8      6          8         8        0              0              0               0    0       0      0 
    7f5ae18cf000 r--p 00027000  fd:00  5509773      4      4      2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/ld-2.27.so
    7f5ae18d0000 rw-p 00028000  fd:00  5509773      4      4      4          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/ld-2.27.so
    7f5ae18d1000 rw-p 00000000  00:00        0      4      4      4          4         4        0              0              0               0    0       0      0 
    7ffea2be0000 rwxp 00000000  00:00        0    128     12     12         12        12        0              0              0               0    0       0      0 [stack]
    7ffea2c00000 rw-p 00000000  00:00        0      4      4      4          4         4        0              0              0               0    0       0      0 
    7ffea2d8c000 r--p 00000000  00:00        0     12      0      0          0         0        0              0              0               0    0       0      0 [vvar]
    7ffea2d8f000 r-xp 00000000  00:00        0      8      4      0          4         0        0              0              0               0    0       0      0 [vdso]
ffffffffff600000 r-xp 00000000  00:00        0      4      0      0          0         0        0              0              0               0    0       0      0 [vsyscall]
                                               ====== ====== ====== ========== ========= ======== ============== ============== =============== ==== ======= ====== 
                                               295020 176132 171431     175124    171168        0              0              0               0    0       0      0 KB 

Initiate 60,000 concurrent connections using wrk2

~/wrk2/wrk -t4 -c60000 -d30m -R600 --timeout 5s  --latency http://10.10.10.10:34402/1kb.html

The memory usage of the worker process after concurrent processing of 60,000 connections

top - 00:33:33 up 7 days, 14:56, 13 users,  load average: 0.06, 0.18, 0.17
任务:   2 total,   0 running,   2 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.5 us,  2.0 sy,  0.0 ni, 93.3 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem : 16316024 total,   947040 free,  7199648 used,  8169336 buff/cache
KiB Swap:  1003516 total,   866044 free,   137472 used.  7331680 avail Mem

进USER      PR  NI    VIRT    RES    SHR  %CPU %MEM     TIME+ COMMAND
 9483 root      20   0 2961852 1.939g   8144 S   0.0 12.5   4:31.12 nginx
 9482 root      20   0  121024   1536      0 S   0.0  0.0   0:00.00 nginx

pmap statistics when the worker process after concurrent processing of 60,000 connections

9483:   nginx: worker process                                                               
          住址 Perm   Offset Device    Inode    Size     Rss     Pss Referenced Anonymous LazyFree ShmemPmdMapped Shared_Hugetlb Private_Hugetlb Swap SwapPss Locked Mapping
        4034c000 rw-p 00000000  00:00        0     128      60      30          0        60        0              0              0               0    0       0      0 
    556dc6e93000 r-xp 00000000  fd:00  2621849    3048    1280    1120       1280         0        0              0              0               0    0       0      0 /kssl/TRP/nginx/sbin/nginx
    556dc738c000 r--p 002f9000  fd:00  2621849       8       8       4          8         8        0              0              0               0    0       0      0 /kssl/TRP/nginx/sbin/nginx
    556dc738e000 rw-p 002fb000  fd:00  2621849     128     128      68        108       128        0              0              0               0    0       0      0 /kssl/TRP/nginx/sbin/nginx
    556dc73ae000 rw-p 00000000  00:00        0     144      24      24         24        24        0              0              0               0    0       0      0 
    556dc7a5b000 rw-p 00000000  00:00        0     536     500     348        256       500        0              0              0               0    0       0      0 [heap]
    556dc7ae1000 rw-p 00000000  00:00        0 2666836 1854184 1854184    1854184   1854184        0              0              0               0    0       0      0 [heap]
    7f5acfad8000 rw-p 00000000  00:00        0  169612  169600  169600     169600    169600        0              0              0               0    0       0      0 
    7f5ada07b000 r-xp 00000000  fd:00  5509942      28      28       0         28         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/librt-2.27.so
    7f5ada082000 ---p 00007000  fd:00  5509942    2044       0       0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/librt-2.27.so
    7f5ada281000 r--p 00006000  fd:00  5509942       4       4       4          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/librt-2.27.so
    7f5ada282000 rw-p 00007000  fd:00  5509942       4       4       4          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/librt-2.27.so
    7f5ada283000 r-xp 00000000  fd:00  5505255     240      64      12         64         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_systemd.so.2
    7f5ada2bf000 ---p 0003c000  fd:00  5505255    2044       0       0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_systemd.so.2
    7f5ada4be000 r--p 0003b000  fd:00  5505255      12      12      12         12        12        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_systemd.so.2
    7f5ada4c1000 rw-p 0003e000  fd:00  5505255       4       4       4          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_systemd.so.2
    7f5ada4c2000 rw-s 00000000  00:05 22171283   16384     196     196        196         0        0              0              0               0    0       0      0 /dev/zero (deleted)
    7f5adb4c2000 r-xp 00000000  fd:00  5509891      44      44       0         44         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_files-2.27.so
    7f5adb4cd000 ---p 0000b000  fd:00  5509891    2044       0       0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_files-2.27.so
    7f5adb6cc000 r--p 0000a000  fd:00  5509891       4       4       2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_files-2.27.so
    7f5adb6cd000 rw-p 0000b000  fd:00  5509891       4       4       2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_files-2.27.so
    7f5adb6ce000 rw-p 00000000  00:00        0      24       0       0          0         0        0              0              0               0    0       0      0 
    7f5adb6d4000 r-xp 00000000  fd:00  5509885      92      64       1         64         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnsl-2.27.so
    7f5adb6eb000 ---p 00017000  fd:00  5509885    2044       0       0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnsl-2.27.so
    7f5adb8ea000 r--p 00016000  fd:00  5509885       4       4       2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnsl-2.27.so
    7f5adb8eb000 rw-p 00017000  fd:00  5509885       4       4       2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnsl-2.27.so
    7f5adb8ec000 rw-p 00000000  00:00        0       8       0       0          0         0        0              0              0               0    0       0      0 
    7f5adb8ee000 r-xp 00000000  fd:00  5509902      44      44       1         44         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_nis-2.27.so
    7f5adb8f9000 ---p 0000b000  fd:00  5509902    2044       0       0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_nis-2.27.so
    7f5adbaf8000 r--p 0000a000  fd:00  5509902       4       4       2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_nis-2.27.so
    7f5adbaf9000 rw-p 0000b000  fd:00  5509902       4       4       2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_nis-2.27.so
    7f5adbafa000 r-xp 00000000  fd:00  5509887      32      32       0         32         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_compat-2.27.so
    7f5adbb02000 ---p 00008000  fd:00  5509887    2048       0       0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_compat-2.27.so
    7f5adbd02000 r--p 00008000  fd:00  5509887       4       4       2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_compat-2.27.so
    7f5adbd03000 rw-p 00009000  fd:00  5509887       4       4       4          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libnss_compat-2.27.so
    7f5adbd04000 r-xp 00000000  fd:00  8396806      28      28       0         28         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
    7f5adbd0b000 ---p 00007000  fd:00  8396806    2044       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
    7f5adbf0a000 r--p 00006000  fd:00  8396806       4       4       2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
    7f5adbf0b000 rw-p 00007000  fd:00  8396806       4       4       2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
    7f5adbf0c000 r-xp 00000000  fd:00  8397004    1040      80       4         80         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
    7f5adc010000 ---p 00104000  fd:00  8397004    2044       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
    7f5adc20f000 r--p 00103000  fd:00  8397004      12      12       6          4        12        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
    7f5adc212000 rw-p 00106000  fd:00  8397004       8       8       4          0         8        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
    7f5adc214000 rw-p 00000000  00:00        0       4       0       0          0         0        0              0              0               0    0       0      0 
    7f5adc215000 r-xp 00000000  fd:00  8397052     280      64       4         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
    7f5adc25b000 ---p 00046000  fd:00  8397052    2044       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
    7f5adc45a000 r--p 00045000  fd:00  8397052      12      12       6          4        12        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
    7f5adc45d000 rw-p 00048000  fd:00  8397052       4       4       2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
    7f5adc45e000 rw-p 00000000  00:00        0       4       0       0          0         0        0              0              0               0    0       0      0 
    7f5adc45f000 r-xp 00000000  fd:00  8397036      56      56       4         56         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
    7f5adc46d000 ---p 0000e000  fd:00  8397036    2044       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
    7f5adc66c000 r--p 0000d000  fd:00  8397036       4       4       2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
    7f5adc66d000 rw-p 0000e000  fd:00  8397036       4       4       2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
    7f5adc66e000 r-xp 00000000  fd:00  8397612     160      12       0         12         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
    7f5adc696000 ---p 00028000  fd:00  8397612    2044       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
    7f5adc895000 r--p 00027000  fd:00  8397612       4       4       2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
    7f5adc896000 rw-p 00028000  fd:00  8397612       4       4       2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
    7f5adc897000 r-xp 00000000  fd:00  8396906     508      64       2         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.2
    7f5adc916000 ---p 0007f000  fd:00  8396906    2048       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.2
    7f5adcb16000 r--p 0007f000  fd:00  8396906       4       4       2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.2
    7f5adcb17000 rw-p 00080000  fd:00  8396906       4       4       2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.2
    7f5adcb18000 r-xp 00000000  fd:00  8397040     204      64       2         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.4
    7f5adcb4b000 ---p 00033000  fd:00  8397040    2044       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.4
    7f5adcd4a000 r--p 00032000  fd:00  8397040       4       4       2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.4
    7f5adcd4b000 rw-p 00033000  fd:00  8397040       4       4       2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.4
    7f5adcd4c000 r-xp 00000000  fd:00  8397242     208      76       7         76         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libnettle.so.6.4
    7f5adcd80000 ---p 00034000  fd:00  8397242    2044       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libnettle.so.6.4
    7f5adcf7f000 r--p 00033000  fd:00  8397242       8       8       4          4         8        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libnettle.so.6.4
    7f5adcf81000 rw-p 00035000  fd:00  8397242       4       4       2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libnettle.so.6.4
    7f5adcf82000 r-xp 00000000  fd:00  8397496      68      64       3         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.5
    7f5adcf93000 ---p 00011000  fd:00  8397496    2048       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.5
    7f5add193000 r--p 00011000  fd:00  8397496       4       4       2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.5
    7f5add194000 rw-p 00012000  fd:00  8397496       4       4       2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.5
    7f5add195000 r-xp 00000000  fd:00  8397166    1512      64       2         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libunistring.so.2.1.0
    7f5add30f000 ---p 0017a000  fd:00  8397166    2048       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libunistring.so.2.1.0
    7f5add50f000 r--p 0017a000  fd:00  8397166      12      12       6          4        12        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libunistring.so.2.1.0
    7f5add512000 rw-p 0017d000  fd:00  8397166       4       4       2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libunistring.so.2.1.0
    7f5add513000 r-xp 00000000  fd:00  8397082     112      64       3         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.3
    7f5add52f000 ---p 0001c000  fd:00  8397082    2044       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.3
    7f5add72e000 r--p 0001b000  fd:00  8397082       4       4       2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.3
    7f5add72f000 rw-p 0001c000  fd:00  8397082       4       4       2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.3
    7f5add730000 r-xp 00000000  fd:00  8397280    1128     128       8        128         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
    7f5add84a000 ---p 0011a000  fd:00  8397280    2048       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
    7f5adda4a000 r--p 0011a000  fd:00  8397280      40      40      20          4        40        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
    7f5adda54000 rw-p 00124000  fd:00  8397280      40      40      22          4        40        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
    7f5adda5e000 rw-p 00000000  00:00        0       4       4       2          0         4        0              0              0               0    0       0      0 
    7f5adda5f000 r-xp 00000000  fd:00  8397399      84      68       9         68         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
    7f5adda74000 ---p 00015000  fd:00  8397399    2044       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
    7f5addc73000 r--p 00014000  fd:00  8397399       4       4       2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
    7f5addc74000 rw-p 00015000  fd:00  8397399       4       4       2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
    7f5addc75000 r-xp 00000000  fd:00  8397034     204      64       4         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
    7f5addca8000 ---p 00033000  fd:00  8397034    2044       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
    7f5addea7000 r--p 00032000  fd:00  8397034       8       8       4          4         8        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
    7f5addea9000 rw-p 00034000  fd:00  8397034       4       4       2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
    7f5addeaa000 rw-p 00000000  00:00        0       4       0       0          0         0        0              0              0               0    0       0      0 
    7f5addeab000 r-xp 00000000  fd:00  5505070      12      12       1         12         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libcom_err.so.2.1
    7f5addeae000 ---p 00003000  fd:00  5505070    2044       0       0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libcom_err.so.2.1
    7f5ade0ad000 r--p 00002000  fd:00  5505070       4       4       2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libcom_err.so.2.1
    7f5ade0ae000 rw-p 00003000  fd:00  5505070       4       4       2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libcom_err.so.2.1
    7f5ade0af000 r-xp 00000000  fd:00  8396540     632     124       9        124         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
    7f5ade14d000 ---p 0009e000  fd:00  8396540    2048       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
    7f5ade34d000 r--p 0009e000  fd:00  8396540       4       4       2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
    7f5ade34e000 rw-p 0009f000  fd:00  8396540      12      12       6          0        12        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
    7f5ade351000 r-xp 00000000  fd:00  8397137     540     124       9        124         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
    7f5ade3d8000 ---p 00087000  fd:00  8397137    2044       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
    7f5ade5d7000 r--p 00086000  fd:00  8397137      16      16       8          4        16        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
    7f5ade5db000 rw-p 0008a000  fd:00  8397137       8       8       4          0         8        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
    7f5ade5dd000 rw-p 00000000  00:00        0       4       0       0          0         0        0              0              0               0    0       0      0 
    7f5ade5de000 r-xp 00000000  fd:00  8397038      32      32       2         32         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
    7f5ade5e6000 ---p 00008000  fd:00  8397038    2044       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
    7f5ade7e5000 r--p 00007000  fd:00  8397038       4       4       2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
    7f5ade7e6000 rw-p 00008000  fd:00  8397038       4       4       2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
    7f5ade7e7000 r-xp 00000000  fd:00  8405969    1376     164       8        164         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.14.10
    7f5ade93f000 ---p 00158000  fd:00  8405969    2044       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.14.10
    7f5adeb3e000 r--p 00157000  fd:00  8405969      48      48      24          4        48        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.14.10
    7f5adeb4a000 rw-p 00163000  fd:00  8405969       4       4       4          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.14.10
    7f5adeb4b000 rw-p 00000000  00:00        0       4       4       2          0         4        0              0              0               0    0       0      0 
    7f5adeb4c000 r-xp 00000000  fd:00  8396961     244      64       4         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
    7f5adeb89000 ---p 0003d000  fd:00  8396961    2048       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
    7f5aded89000 r--p 0003d000  fd:00  8396961       8       8       4          4         8        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
    7f5aded8b000 rw-p 0003f000  fd:00  8396961       8       8       4          0         8        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
    7f5aded8d000 r-xp 00000000  fd:00  8397423     100      64       6         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
    7f5adeda6000 ---p 00019000  fd:00  8397423    2048       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
    7f5adefa6000 r--p 00019000  fd:00  8397423       4       4       2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
    7f5adefa7000 rw-p 0001a000  fd:00  8397423       4       4       2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
    7f5adefa8000 r-xp 00000000  fd:00  5509940      92      60       0         60         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libresolv-2.27.so
    7f5adefbf000 ---p 00017000  fd:00  5509940    2048       0       0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libresolv-2.27.so
    7f5adf1bf000 r--p 00017000  fd:00  5509940       4       4       2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libresolv-2.27.so
    7f5adf1c0000 rw-p 00018000  fd:00  5509940       4       4       2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libresolv-2.27.so
    7f5adf1c1000 rw-p 00000000  00:00        0       8       0       0          0         0        0              0              0               0    0       0      0 
    7f5adf1c3000 r-xp 00000000  fd:00  8389082      52      52       3         52         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.8
    7f5adf1d0000 ---p 0000d000  fd:00  8389082    2044       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.8
    7f5adf3cf000 r--p 0000c000  fd:00  8389082       4       4       2          4         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.8
    7f5adf3d0000 rw-p 0000d000  fd:00  8389082       4       4       2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.8
    7f5adf3d1000 r-xp 00000000  fd:00  5505267      92      64       3         64         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f5adf3e8000 ---p 00017000  fd:00  5505267    2044       0       0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f5adf5e7000 r--p 00016000  fd:00  5505267       4       4       2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f5adf5e8000 rw-p 00017000  fd:00  5505267       4       4       2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f5adf5e9000 r-xp 00000000  fd:00  8389083     312      64       4         64         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8
    7f5adf637000 ---p 0004e000  fd:00  8389083    2044       0       0          0         0        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8
    7f5adf836000 r--p 0004d000  fd:00  8389083       8       8       4          4         8        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8
    7f5adf838000 rw-p 0004f000  fd:00  8389083       4       4       2          0         4        0              0              0               0    0       0      0 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8
    7f5adf839000 rw-p 00000000  00:00        0       8       0       0          0         0        0              0              0               0    0       0      0 
    7f5adf83b000 r-xp 00000000  fd:00  5509801    1948    1552       8       1552         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libc-2.27.so
    7f5adfa22000 ---p 001e7000  fd:00  5509801    2048       0       0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libc-2.27.so
    7f5adfc22000 r--p 001e7000  fd:00  5509801      16      16       8          8        16        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libc-2.27.so
    7f5adfc26000 rw-p 001eb000  fd:00  5509801       8       8       8          8         8        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libc-2.27.so
    7f5adfc28000 rw-p 00000000  00:00        0      16      16      16         16        16        0              0              0               0    0       0      0 
    7f5adfc2c000 r-xp 00000000  fd:00  5509973     112      64       0         64         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libz.so.1.2.11
    7f5adfc48000 ---p 0001c000  fd:00  5509973    2044       0       0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libz.so.1.2.11
    7f5adfe47000 r--p 0001b000  fd:00  5509973       4       4       2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libz.so.1.2.11
    7f5adfe48000 rw-p 0001c000  fd:00  5509973       4       4       2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libz.so.1.2.11
    7f5adfe49000 r-xp 00000000  fd:00  5509923     448      64       0         64         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libpcre.so.3.13.3
    7f5adfeb9000 ---p 00070000  fd:00  5509923    2048       0       0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libpcre.so.3.13.3
    7f5ae00b9000 r--p 00070000  fd:00  5509923       4       4       2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libpcre.so.3.13.3
    7f5ae00ba000 rw-p 00071000  fd:00  5509923       4       4       2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libpcre.so.3.13.3
    7f5ae00bb000 r-xp 00000000  fd:00  5509864    1652      84       1         84         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libm-2.27.so
    7f5ae0258000 ---p 0019d000  fd:00  5509864    2044       0       0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libm-2.27.so
    7f5ae0457000 r--p 0019c000  fd:00  5509864       4       4       2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libm-2.27.so
    7f5ae0458000 rw-p 0019d000  fd:00  5509864       4       4       2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libm-2.27.so
    7f5ae0459000 r-xp 00000000  fd:00  2621865     620      64      64         64         0        0              0              0               0    0       0      0 /kssl/TRP/luajit/lib/libluajit-5.1.so.2.1.0
    7f5ae04f4000 ---p 0009b000  fd:00  2621865    2044       0       0          0         0        0              0              0               0    0       0      0 /kssl/TRP/luajit/lib/libluajit-5.1.so.2.1.0
    7f5ae06f3000 r--p 0009a000  fd:00  2621865       8       8       4          4         8        0              0              0               0    0       0      0 /kssl/TRP/luajit/lib/libluajit-5.1.so.2.1.0
    7f5ae06f5000 rw-p 0009c000  fd:00  2621865       4       4       2          0         4        0              0              0               0    0       0      0 /kssl/TRP/luajit/lib/libluajit-5.1.so.2.1.0
    7f5ae06f6000 r-xp 00000000  fd:00  5509811      36      36       1         36         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libcrypt-2.27.so
    7f5ae06ff000 ---p 00009000  fd:00  5509811    2044       0       0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libcrypt-2.27.so
    7f5ae08fe000 r--p 00008000  fd:00  5509811       4       4       2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libcrypt-2.27.so
    7f5ae08ff000 rw-p 00009000  fd:00  5509811       4       4       2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libcrypt-2.27.so
    7f5ae0900000 rw-p 00000000  00:00        0     184       0       0          0         0        0              0              0               0    0       0      0 
    7f5ae092e000 r-xp 00000000  fd:00  5509934     104     104       0        104         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libpthread-2.27.so
    7f5ae0948000 ---p 0001a000  fd:00  5509934    2044       0       0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libpthread-2.27.so
    7f5ae0b47000 r--p 00019000  fd:00  5509934       4       4       2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libpthread-2.27.so
    7f5ae0b48000 rw-p 0001a000  fd:00  5509934       4       4       4          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libpthread-2.27.so
    7f5ae0b49000 rw-p 00000000  00:00        0      16       4       4          4         4        0              0              0               0    0       0      0 
    7f5ae0b4d000 r-xp 00000000  fd:00  5509824      12      12       0         12         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libdl-2.27.so
    7f5ae0b50000 ---p 00003000  fd:00  5509824    2044       0       0          0         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libdl-2.27.so
    7f5ae0d4f000 r--p 00002000  fd:00  5509824       4       4       2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libdl-2.27.so
    7f5ae0d50000 rw-p 00003000  fd:00  5509824       4       4       2          0         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/libdl-2.27.so
    7f5ae0d51000 r-xp 00000000  fd:00  9309093      32      28      28         28         0        0              0              0               0    0       0      0 /usr/local/ssl/lib/libKLCRL.so.1.0.0
    7f5ae0d59000 ---p 00008000  fd:00  9309093    2044       0       0          0         0        0              0              0               0    0       0      0 /usr/local/ssl/lib/libKLCRL.so.1.0.0
    7f5ae0f58000 r--p 00007000  fd:00  9309093       4       4       2          4         4        0              0              0               0    0       0      0 /usr/local/ssl/lib/libKLCRL.so.1.0.0
    7f5ae0f59000 rw-p 00008000  fd:00  9309093       4       4       2          0         4        0              0              0               0    0       0      0 /usr/local/ssl/lib/libKLCRL.so.1.0.0
    7f5ae0f5a000 r-xp 00000000  fd:00  9309081    2564    1992     452       1992         0        0              0              0               0    0       0      0 /usr/local/ssl/lib/libcrypto.so.1.0.0
    7f5ae11db000 ---p 00281000  fd:00  9309081    2044       0       0          0         0        0              0              0               0    0       0      0 /usr/local/ssl/lib/libcrypto.so.1.0.0
    7f5ae13da000 r--p 00280000  fd:00  9309081     152     140      70        104       140        0              0              0               0    0       0      0 /usr/local/ssl/lib/libcrypto.so.1.0.0
    7f5ae1400000 rw-p 002a6000  fd:00  9309081      48      48      26         36        48        0              0              0               0    0       0      0 /usr/local/ssl/lib/libcrypto.so.1.0.0
    7f5ae140c000 rw-p 00000000  00:00        0      16      16      10         12        16        0              0              0               0    0       0      0 
    7f5ae1410000 r-xp 00000000  fd:00  9309084     560     508     508        508         0        0              0              0               0    0       0      0 /usr/local/ssl/lib/libssl.so.1.0.0
    7f5ae149c000 ---p 0008c000  fd:00  9309084    2048       0       0          0         0        0              0              0               0    0       0      0 /usr/local/ssl/lib/libssl.so.1.0.0
    7f5ae169c000 r--p 0008c000  fd:00  9309084      16      16       8          8        16        0              0              0               0    0       0      0 /usr/local/ssl/lib/libssl.so.1.0.0
    7f5ae16a0000 rw-p 00090000  fd:00  9309084      28      28      16         24        28        0              0              0               0    0       0      0 /usr/local/ssl/lib/libssl.so.1.0.0
    7f5ae16a7000 rw-p 00000000  00:00        0       4       4       2          0         4        0              0              0               0    0       0      0 
    7f5ae16a8000 r-xp 00000000  fd:00  5509773     156     156       0        156         0        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/ld-2.27.so
    7f5ae189b000 rw-p 00000000  00:00        0      68      64      38         56        64        0              0              0               0    0       0      0 
    7f5ae18cc000 rw-s 00000000  00:05 22171285       4       4       4          4         0        0              0              0               0    0       0      0 /dev/zero (deleted)
    7f5ae18cd000 rw-p 00000000  00:00        0       8       8       6          8         8        0              0              0               0    0       0      0 
    7f5ae18cf000 r--p 00027000  fd:00  5509773       4       4       2          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/ld-2.27.so
    7f5ae18d0000 rw-p 00028000  fd:00  5509773       4       4       4          4         4        0              0              0               0    0       0      0 /lib/x86_64-linux-gnu/ld-2.27.so
    7f5ae18d1000 rw-p 00000000  00:00        0       4       4       4          4         4        0              0              0               0    0       0      0 
    7ffea2be0000 rwxp 00000000  00:00        0     128      16      16         16        16        0              0              0               0    0       0      0 [stack]
    7ffea2c00000 rw-p 00000000  00:00        0       4       4       4          4         4        0              0              0               0    0       0      0 
    7ffea2d8c000 r--p 00000000  00:00        0      12       0       0          0         0        0              0              0               0    0       0      0 [vvar]
    7ffea2d8f000 r-xp 00000000  00:00        0       8       4       0          4         0        0              0              0               0    0       0      0 [vdso]
ffffffffff600000 r-xp 00000000  00:00        0       4       0       0          0         0        0              0              0               0    0       0      0 [vsyscall]
                                               ======= ======= ======= ========== ========= ======== ============== ============== =============== ==== ======= ====== 
                                               2961856 2033568 2027277    2032828   2025424        0              0              0               0    0       0      0 KB 

my nginx configuration is as follows

user  root root;
worker_rlimit_nofile 819200;
worker_rlimit_core 500M;
worker_processes 1;
worker_cpu_affinity auto;
working_directory /tmp/cores/;

error_log  logs/error.log  error;
pid        logs/nginx.pid;

events {
    worker_connections  409600;
}

stream {

    error_log logs/error.log warn;

    ssl_certificate pem/rsa.pem;
    ssl_certificate_key pem/rsa.key;
    ssl_client_certificate pem/ca/rsa_ca.pem;
    ssl_verify_client on;
    ssl_verify_depth 3;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECC-SM4-SM3:ECDHE-SM4-SM3:RSA-SM4-SM3;
    ssl_prefer_server_ciphers on;
    ssl_session_timeout 5m;
    ssl_session_cache  shared:tcp:16m;
    ssl_session_tickets off;

    proxy_connect_timeout 600s;
    proxy_timeout 600m;

    server {
        listen 0.0.0.0:34402 backlog=65535 ssl;

        proxy_protocol off;
        proxy_pass 10.0.10.10:8080;
    }
}

Operating system and platform information

root@ubuntu:/# uname -a
Linux ubuntu 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux


root@ubuntu:/# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.4.0-1ubuntu1~18.04.1' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)


root@ubuntu:/# /kssl/TRP/nginx/sbin/nginx -V
nginx version: nginx/1.13.6.2
built by gcc 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
built with OpenSSL 1.0.2o  27 Mar 2018 (running with OpenSSL 1.0.2o  27 Mar 2018 [])
TLS SNI support enabled
configure arguments: --with-debug --with-cc-opt='-DNGX_LUA_USE_ASSERT -DNGX_LUA_ABORT_AT_PANIC -O2 -g3 -O0 -Wall -Werror -I/usr/local/ssl/include' --with-ld-opt='-Wl,-rpath,/kssl/TRP/luajit/lib -L/usr/local/ssl/lib -lssl -lcrypto' --with-http_sub_module --with-stream_ssl_preread_module --with-ipv6 --http-fastcgi-temp-path=/dev/null --http-uwsgi-temp-path=/dev/null --http-scgi-temp-path=/dev/null --http-client-body-temp-path=logs/client-body-temp --http-proxy-temp-path=logs/proxy-temp --with-stream --with-stream_ssl_module --with-http_ssl_module

Attachments (3)

1.png (82.6 KB) - added by honghaier250@… 3 weeks ago.
2.png (87.6 KB) - added by honghaier250@… 3 weeks ago.
3.png (87.5 KB) - added by honghaier250@… 3 weeks ago.

Download all attachments as: .zip

Change History (15)

comment:1 Changed 5 weeks ago by honghaier250@…

Sorry, correct spelling mistakes

Initiate 60,000 concurrent connections using wrk2

~/wrk2/wrk -t4 -c60000 -d30m -R600 --timeout 5s  --latency https://10.10.10.10:34402/1kb.html

comment:2 Changed 5 weeks ago by mdounin

So, you are seeing heap being large after processing a large number of concurrent connections. This doesn't mean there is a memory leak though - this memory might be perfectly freed by nginx, but held by the memory allocator and not returned to the system. Further, it might not be possible to return memory from heap to the system, as it is only possible to return memory from heap if there are no allocations at higher addresses. Note that such memory, while not returned to the system, is properly freed, and can be used for other allocations, such as for handling other connections.

What do you see if you'll run wrk again, with another 60k connections? Is heap grows again, to 4+ gigabytes? Or it stays roughly at the same level, at ~2 gigabytes?

comment:3 Changed 3 weeks ago by honghaier250@…

Yes, I do run wrk again with 60k connections, heap grows again to 2+ gigabytes. The more times I try, more heap grows again.

comment:4 Changed 3 weeks ago by mdounin

Could you please make sure you are seeing the same behaviour with vanilla nginx, without any 3rd party patches and modules, as available from nginx.org?

comment:5 Changed 3 weeks ago by honghaier250@…

Yes, and I try it again with http://nginx.org/download/nginx-1.17.5.tar.gz, the phenomenon is the same.

➜  /tmp /kssl/TRP/nginx/sbin/nginx -V
nginx version: nginx/1.17.5
built by gcc 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1) 
built with OpenSSL 1.1.1  11 Sep 2018
TLS SNI support enabled
configure arguments: --with-debug --with-cc-opt='-O2 -Wall -Werror' --with-http_sub_module --with-stream_ssl_preread_module --with-ipv6 --http-fastcgi-temp-path=/dev/null --http-uwsgi-temp-path=/dev/null --http-scgi-temp-path=/dev/null --http-client-body-temp-path=logs/client-body-temp --http-proxy-temp-path=logs/proxy-temp --with-stream --with-stream_ssl_module --with-http_ssl_module
➜  /tmp 
➜  /tmp 
➜  /tmp openssl version -a
OpenSSL 1.1.1  11 Sep 2018
built on: Thu Jun 20 17:36:28 2019 UTC
platform: debian-amd64
options:  bn(64,64) rc4(16x,int) des(int) blowfish(ptr) 
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -fdebug-prefix-map=/build/openssl-cn9tZy/openssl-1.1.1=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
OPENSSLDIR: "/usr/lib/ssl"
ENGINESDIR: "/usr/lib/x86_64-linux-gnu/engines-1.1"
Seeding source: os-specific

comment:6 Changed 3 weeks ago by honghaier250@…

Last edited 3 weeks ago by honghaier250@… (previous) (diff)

Changed 3 weeks ago by honghaier250@…

Changed 3 weeks ago by honghaier250@…

Changed 3 weeks ago by honghaier250@…

comment:7 Changed 3 weeks ago by honghaier250@…

The data is recorded as follows

First Time
https://trac.nginx.org/nginx/attachment/ticket/1873/1.png

Second Time
https://trac.nginx.org/nginx/attachment/ticket/1873/2.png

Third Time
https://trac.nginx.org/nginx/attachment/ticket/1873/3.png

comment:8 Changed 3 weeks ago by honghaier250@…

To check if the system can reuse the memory occupied by nginx, I tested it with stress-ng.

➜  ~ sudo stress-ng --vm 2 --vm-bytes 12G --timeout 60s --metrics-brief
stress-ng: info:  [20483] dispatching hogs: 2 vm

stress-ng: error: [20623] stress-ng-vm: gave up trying to mmap, no available memory

stress-ng: info:  [20483] successful run completed in 60.19s (1 min, 0.19 secs)
stress-ng: info:  [20483] stressor       bogo ops real time  usr time  sys time   bogo ops/s   bogo ops/s
stress-ng: info:  [20483]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: info:  [20483] vm              3343348     37.79     51.28     13.05     88476.65     51971.83


➜  ~ sudo stress-ng --vm 2 --vm-bytes 10G --timeout 60s --metrics-brief
stress-ng: info:  [1485] dispatching hogs: 2 vm

stress-ng: error: [1693] stress-ng-vm: gave up trying to mmap, no available memory

stress-ng: info:  [1485] successful run completed in 60.07s (1 min, 0.07 secs)
stress-ng: info:  [1485] stressor       bogo ops real time  usr time  sys time   bogo ops/s   bogo ops/s
stress-ng: info:  [1485]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: info:  [1485] vm              5407604     37.50     52.79     11.74    144187.24     83799.85

After stoped the nginx process, test again using stress-ng

➜  stress-ng-0.10.08 sudo stress-ng --vm 2 --vm-bytes 10G --timeout 60s --metrics-brief
stress-ng: info:  [10461] dispatching hogs: 2 vm
stress-ng: info:  [10461] successful run completed in 60.16s (1 min, 0.16 secs)
stress-ng: info:  [10461] stressor       bogo ops real time  usr time  sys time   bogo ops/s   bogo ops/s
stress-ng: info:  [10461]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: info:  [10461] vm              8193768     60.10    101.90     18.26    136344.17     68190.48


➜  stress-ng-0.10.08 sudo stress-ng --vm 2 --vm-bytes 12G --timeout 60s --metrics-brief
stress-ng: info:  [12658] dispatching hogs: 2 vm
stress-ng: info:  [12658] successful run completed in 60.27s (1 min, 0.27 secs)
stress-ng: info:  [12658] stressor       bogo ops real time  usr time  sys time   bogo ops/s   bogo ops/s
stress-ng: info:  [12658]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: info:  [12658] vm              6686696     60.26     97.42     22.92    110956.08     55565.03
➜  stress-ng-0.10.08 cheat stress
Last edited 3 weeks ago by honghaier250@… (previous) (diff)

comment:9 Changed 3 weeks ago by mdounin

  • Resolution set to invalid
  • Status changed from new to closed

So, clearly there is no memory leak here: in 1.png VIRT is 3.83G, in 2.png - 3.62G, in 3.png - 3.87G. It stays roughly the same (after the second run some memory even returned the system), and the numbers reflects what was described in the comment:2 - memory is held by the system allocator. Clearly there is no additional 2 gigabytes of memory consumed on each run.

Note well that RES does not reflect allocated memory. Rather, it reflects memory actually touched by the process, and it can grow even if no additional memory allocations are made. But even looking at RES clearly shows there is no memory leak, at it grows only slightly on additional runs, that is, stays roughly the same - 2.77G in 1.png, 3.27G in 2.png, and 3.58G in 3.png.

comment:10 follow-up: Changed 3 weeks ago by honghaier250@…

When using the wrk with 60,000 concurrent connection test, I found that the RES value will not exceed 4G (always maintained at around 3.5G), but RES will gradually increase until 3.5G on first a few times test.

However, when the number of concurrent connections exceeds 60,000 (such as 120,000), the RES will grow again. With more and more concurrent connections, nginx process finally invoke oom-killer.

so my question is:

  1. Why RES gradually increase
  2. If RES does not reflect the memory leak, how to determine if there is a memory leak
Last edited 3 weeks ago by honghaier250@… (previous) (diff)

comment:11 Changed 3 weeks ago by honghaier250@…

Oct 29 12:02:50 ubuntu kernel: [80713.796546] nginx invoked oom-killer: gfp_mask=0x14280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null), order=0, oom_score_adj=0
Oct 29 12:02:50 ubuntu kernel: [80713.796547] nginx cpuset=/ mems_allowed=0
Oct 29 12:02:50 ubuntu kernel: [80713.796550] CPU: 0 PID: 29267 Comm: nginx Tainted: G           OE    4.15.0-66-generic #75-Ubuntu
Oct 29 12:02:50 ubuntu kernel: [80713.796551] Hardware name: MSI MS-7A15/H110M PRO-A (MS-7A15), BIOS 1.90 12/01/2017
Oct 29 12:02:50 ubuntu kernel: [80713.796551] Call Trace:
Oct 29 12:02:50 ubuntu kernel: [80713.796557]  dump_stack+0x63/0x8e
Oct 29 12:02:50 ubuntu kernel: [80713.796559]  dump_header+0x71/0x285
Oct 29 12:02:50 ubuntu kernel: [80713.796561]  ? security_capable_noaudit+0x4b/0x70
Oct 29 12:02:50 ubuntu kernel: [80713.796562]  oom_kill_process+0x21f/0x420
Oct 29 12:02:50 ubuntu kernel: [80713.796564]  out_of_memory+0x2b6/0x4d0
Oct 29 12:02:50 ubuntu kernel: [80713.796565]  __alloc_pages_slowpath+0xa53/0xe00
Oct 29 12:02:50 ubuntu kernel: [80713.796567]  ? __switch_to_asm+0x41/0x70
Oct 29 12:02:50 ubuntu kernel: [80713.796569]  __alloc_pages_nodemask+0x29a/0x2c0
Oct 29 12:02:50 ubuntu kernel: [80713.796571]  alloc_pages_vma+0x88/0x1f0
Oct 29 12:02:50 ubuntu kernel: [80713.796573]  __handle_mm_fault+0x8b7/0x1290
Oct 29 12:02:50 ubuntu kernel: [80713.796575]  handle_mm_fault+0xb1/0x210
Oct 29 12:02:50 ubuntu kernel: [80713.796577]  __do_page_fault+0x281/0x4b0
Oct 29 12:02:50 ubuntu kernel: [80713.796578]  do_page_fault+0x2e/0xe0
Oct 29 12:02:50 ubuntu kernel: [80713.796580]  ? page_fault+0x2f/0x50
Oct 29 12:02:50 ubuntu kernel: [80713.796581]  page_fault+0x45/0x50
Oct 29 12:02:50 ubuntu kernel: [80713.796582] RIP: 0033:0x7f6c0827d0dd
Oct 29 12:02:50 ubuntu kernel: [80713.796583] RSP: 002b:00007ffe13584110 EFLAGS: 00010206
Oct 29 12:02:50 ubuntu kernel: [80713.796584] RAX: 0000000000013da1 RBX: 0000000000000341 RCX: 000055d9b01a7f20
Oct 29 12:02:50 ubuntu kernel: [80713.796585] RDX: 0000000000000341 RSI: 0000000000000000 RDI: 000055d9b01a8260
Oct 29 12:02:50 ubuntu kernel: [80713.796585] RBP: ffffffffffffff90 R08: 0000000000000004 R09: 0000000000000000
Oct 29 12:02:50 ubuntu kernel: [80713.796586] R10: 000055d805179010 R11: 0000000000000000 R12: 0000000000000032
Oct 29 12:02:50 ubuntu kernel: [80713.796586] R13: 00007f6c085d3ca0 R14: 00007f6c085d3c40 R15: 0000000000000000
Oct 29 12:02:50 ubuntu kernel: [80713.796587] Mem-Info:
Oct 29 12:02:50 ubuntu kernel: [80713.796590] active_anon:2083802 inactive_anon:287981 isolated_anon:0
Oct 29 12:02:50 ubuntu kernel: [80713.796590]  active_file:1780 inactive_file:2175 isolated_file:20
Oct 29 12:02:50 ubuntu kernel: [80713.796590]  unevictable:112 dirty:0 writeback:0 unstable:0
Oct 29 12:02:50 ubuntu kernel: [80713.796590]  slab_reclaimable:69289 slab_unreclaimable:220913
Oct 29 12:02:50 ubuntu kernel: [80713.796590]  mapped:1385736 shmem:120827 pagetables:28904 bounce:0
Oct 29 12:02:50 ubuntu kernel: [80713.796590]  free:33273 free_pcp:5 free_cma:0
Oct 29 12:02:50 ubuntu kernel: [80713.796592] Node 0 active_anon:8335208kB inactive_anon:1151924kB active_file:7120kB inactive_file:8700kB unevictable:448kB isolated(anon):0kB isolated(file):80kB mapped:5542944kB dirty:0kB writeback:0kB shmem:483308kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Oct 29 12:02:50 ubuntu kernel: [80713.796592] Node 0 DMA free:15896kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB managed:15896kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct 29 12:02:50 ubuntu kernel: [80713.796594] lowmem_reserve[]: 0 2064 15871 15871 15871
Oct 29 12:02:50 ubuntu kernel: [80713.796596] Node 0 DMA32 free:58528kB min:8780kB low:10972kB high:13164kB active_anon:21268kB inactive_anon:44kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:2249240kB managed:2162080kB mlocked:0kB kernel_stack:32kB pagetables:156kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct 29 12:02:50 ubuntu kernel: [80713.796598] lowmem_reserve[]: 0 0 13806 13806 13806
Oct 29 12:02:50 ubuntu kernel: [80713.796600] Node 0 Normal free:58668kB min:58732kB low:73412kB high:88092kB active_anon:8313856kB inactive_anon:1151880kB active_file:6744kB inactive_file:8604kB unevictable:448kB writepending:0kB present:14401536kB managed:14138036kB mlocked:448kB kernel_stack:21760kB pagetables:115460kB bounce:0kB free_pcp:20kB local_pcp:0kB free_cma:0kB
Oct 29 12:02:50 ubuntu kernel: [80713.796602] lowmem_reserve[]: 0 0 0 0 0
Oct 29 12:02:50 ubuntu kernel: [80713.796604] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15896kB
Oct 29 12:02:50 ubuntu kernel: [80713.796610] Node 0 DMA32: 118*4kB (UMEH) 85*8kB (UMEH) 50*16kB (UMEH) 278*32kB (M) 179*64kB (M) 81*128kB (M) 43*256kB (ME) 25*512kB (ME) 2*1024kB (M) 0*2048kB 0*4096kB = 58528kB
Oct 29 12:02:50 ubuntu kernel: [80713.796617] Node 0 Normal: 550*4kB (UMEH) 183*8kB (UMEH) 3480*16kB (UMH) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59344kB
Oct 29 12:02:50 ubuntu kernel: [80713.796623] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Oct 29 12:02:50 ubuntu kernel: [80713.796624] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 29 12:02:50 ubuntu kernel: [80713.796624] 127891 total pagecache pages
Oct 29 12:02:50 ubuntu kernel: [80713.796625] 3075 pages in swap cache
Oct 29 12:02:50 ubuntu kernel: [80713.796626] Swap cache stats: add 317762, delete 314678, find 52323/64083
Oct 29 12:02:50 ubuntu kernel: [80713.796626] Free swap  = 0kB
Oct 29 12:02:50 ubuntu kernel: [80713.796627] Total swap = 1003516kB
Oct 29 12:02:50 ubuntu kernel: [80713.796627] 4166689 pages RAM
Oct 29 12:02:50 ubuntu kernel: [80713.796627] 0 pages HighMem/MovableOnly
Oct 29 12:02:50 ubuntu kernel: [80713.796628] 87686 pages reserved
Oct 29 12:02:50 ubuntu kernel: [80713.796628] 0 pages cma reserved
Oct 29 12:02:50 ubuntu kernel: [80713.796628] 0 pages hwpoisoned
Oct 29 12:02:50 ubuntu kernel: [80713.796629] [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Oct 29 12:02:50 ubuntu kernel: [80713.796633] [  526]     0   526    30013      349   262144        0             0 systemd-journal
Oct 29 12:02:50 ubuntu kernel: [80713.796635] [  545]     0   545    26476        1   106496       56             0 lvmetad
Oct 29 12:02:50 ubuntu kernel: [80713.796636] [  560]     0   560    12070      164   131072      682         -1000 systemd-udevd
Oct 29 12:02:50 ubuntu kernel: [80713.796639] [  662] 62583   662    36528       40   188416      116             0 systemd-timesyn
Oct 29 12:02:50 ubuntu kernel: [80713.796640] [  663]   101   663    17690      107   172032       89             0 systemd-resolve
Oct 29 12:02:50 ubuntu kernel: [80713.796641] [  664]     0   664    11900       53   135168       45             0 rpcbind
Oct 29 12:02:50 ubuntu kernel: [80713.796642] [ 1260]   102  1260   108909      187   237568      209             0 rsyslogd
Oct 29 12:02:50 ubuntu kernel: [80713.796644] [ 1268]     0  1268    17684       81   184320      128             0 systemd-logind
Oct 29 12:02:50 ubuntu kernel: [80713.796645] [ 1275]     0  1275    27620       40   114688       44             0 irqbalance
Oct 29 12:02:50 ubuntu kernel: [80713.796646] [ 1285]     0  1285   125972      647   352256      278             0 udisksd
Oct 29 12:02:50 ubuntu kernel: [80713.796647] [ 1289]     0  1289     9607       40   114688       33             0 cron
Oct 29 12:02:50 ubuntu kernel: [80713.796648] [ 1291]   103  1291    12946      435   143360      175          -900 dbus-daemon
Oct 29 12:02:50 ubuntu kernel: [80713.796649] [ 1330]     0  1330    11307       70   126976       68             0 wpa_supplicant
Oct 29 12:02:50 ubuntu kernel: [80713.796650] [ 1341]   116  1341    11863      107   135168       57             0 avahi-daemon
Oct 29 12:02:50 ubuntu kernel: [80713.796651] [ 1343]     0  1343   108581      183   348160      172             0 ModemManager
Oct 29 12:02:50 ubuntu kernel: [80713.796652] [ 1348]     0  1348    74297      401   204800      425             0 accounts-daemon
Oct 29 12:02:50 ubuntu kernel: [80713.796653] [ 1352]     0  1352   325961     2057   315392      922          -900 snapd
Oct 29 12:02:50 ubuntu kernel: [80713.796654] [ 1361]     0  1361     1138        0    57344       17             0 acpid
Oct 29 12:02:50 ubuntu kernel: [80713.796655] [ 1362]     0  1362   123173      458   446464      395             0 NetworkManager
Oct 29 12:02:50 ubuntu kernel: [80713.796656] [ 1369]     0  1369    46742      120   200704       84             0 thermald
Oct 29 12:02:50 ubuntu kernel: [80713.796657] [ 1371]     0  1371    44409      440   237568     1557             0 networkd-dispat
Oct 29 12:02:50 ubuntu kernel: [80713.796658] [ 1380]     0  1380    75934      486   225280      374             0 polkitd
Oct 29 12:02:50 ubuntu kernel: [80713.796659] [ 1382]   116  1382    11769       14   126976       72             0 avahi-daemon
Oct 29 12:02:50 ubuntu kernel: [80713.796660] [ 1587]     0  1587   219841      386   208896       97             0 canonical-livep
Oct 29 12:02:50 ubuntu kernel: [80713.796661] [ 1594]     0  1594    48585      464   266240     1513             0 unattended-upgr
Oct 29 12:02:50 ubuntu kernel: [80713.796661] [ 1598]   127  1598    19267      285   188416      890             0 snmpd
Oct 29 12:02:50 ubuntu kernel: [80713.796662] [ 1627]   128  1627    43603      100   106496       81             0 privoxy
Oct 29 12:02:50 ubuntu kernel: [80713.796663] [ 1630]     0  1630     5862      129    81920       67             0 EasyMonitor
Oct 29 12:02:50 ubuntu kernel: [80713.796664] [ 1634]     0  1634   356289     3425   364544     1431             0 containerd
Oct 29 12:02:50 ubuntu kernel: [80713.796665] [ 1642]     0  1642    77075      129   225280      170             0 gdm3
Oct 29 12:02:50 ubuntu kernel: [80713.796666] [ 1643]     0  1643    18235       19   180224      171         -1000 sshd
Oct 29 12:02:50 ubuntu kernel: [80713.796667] [ 1665]     0  1665    64968       85   139264       90             0 ECAgent
Oct 29 12:02:50 ubuntu kernel: [80713.796668] [ 1670]     0  1670    65501      585   290816      448             0 gdm-session-wor
Oct 29 12:02:50 ubuntu kernel: [80713.796669] [ 1763]   121  1763    19289      162   192512      234             0 systemd
Oct 29 12:02:50 ubuntu kernel: [80713.796670] [ 1768]   121  1768    28597       29   262144      708             0 (sd-pam)
Oct 29 12:02:50 ubuntu kernel: [80713.796671] [ 1803]   121  1803    49479        0   147456      143             0 gdm-wayland-ses
Oct 29 12:02:50 ubuntu kernel: [80713.796672] [ 1805]   121  1805    12565        3   139264      221             0 dbus-daemon
Oct 29 12:02:50 ubuntu kernel: [80713.796673] [ 1816]   121  1816   158254      166   430080      429             0 gnome-session-b
Oct 29 12:02:50 ubuntu kernel: [80713.796674] [ 1838]   121  1838   855299    16274  1413120     7874             0 gnome-shell
Oct 29 12:02:50 ubuntu kernel: [80713.796675] [ 1851]   124  1851     1131       11    57344        5             0 uml_switch
Oct 29 12:02:50 ubuntu kernel: [80713.796676] [ 1867]     0  1867    80599      134   245760      188             0 upowerd
Oct 29 12:02:50 ubuntu kernel: [80713.796677] [ 1979]   121  1979   153653       53   561152     3031             0 Xwayland
Oct 29 12:02:50 ubuntu kernel: [80713.796678] [ 1996]   121  1996    87320        1   188416      196             0 at-spi-bus-laun
Oct 29 12:02:50 ubuntu kernel: [80713.796679] [ 2001]   121  2001    12482        0   139264      119             0 dbus-daemon
Oct 29 12:02:50 ubuntu kernel: [80713.796680] [ 2003]   121  2003    55191        2   196608      204             0 at-spi2-registr
Oct 29 12:02:50 ubuntu kernel: [80713.796681] [ 2007]   121  2007   270523       53   372736      742             0 pulseaudio
Oct 29 12:02:50 ubuntu kernel: [80713.796682] [ 2008]   109  2008    45876       20   126976       47             0 rtkit-daemon
Oct 29 12:02:50 ubuntu kernel: [80713.796683] [ 2057]   112  2057   115548       99   401408      316             0 whoopsie
Oct 29 12:02:50 ubuntu kernel: [80713.796684] [ 2060]     0  2060   324267     4763   548864     3869          -500 dockerd
Oct 29 12:02:50 ubuntu kernel: [80713.796685] [ 2065]   113  2065    14235       28   143360       86             0 kerneloops
Oct 29 12:02:50 ubuntu kernel: [80713.796686] [ 2075]   113  2075    14235       40   147456       74             0 kerneloops
Oct 29 12:02:50 ubuntu kernel: [80713.796687] [ 2240]   121  2240    90432        3   192512      368             0 ibus-daemon
Oct 29 12:02:50 ubuntu kernel: [80713.796688] [ 2272]   121  2272    70246        0   188416      181             0 ibus-dconf
Oct 29 12:02:50 ubuntu kernel: [80713.796689] [ 2275]   121  2275   122178        0   569344     1800             0 ibus-x11
Oct 29 12:02:50 ubuntu kernel: [80713.796690] [ 2277]   121  2277    69699        0   176128      172             0 ibus-portal
Oct 29 12:02:50 ubuntu kernel: [80713.796690] [ 2471]     0  2471    74270      125   212992       87             0 boltd
Oct 29 12:02:50 ubuntu kernel: [80713.796691] [ 2484]     0  2484   112281     2529   368640     5694             0 packagekitd
Oct 29 12:02:50 ubuntu kernel: [80713.796692] [ 2502]   121  2502   159894        4   598016     1972             0 gsd-xsettings
Oct 29 12:02:50 ubuntu kernel: [80713.796693] [ 2505]   121  2505    69612        0   176128      177             0 gsd-a11y-settin
Oct 29 12:02:50 ubuntu kernel: [80713.796694] [ 2512]   121  2512   122094        3   577536     1802             0 gsd-clipboard
Oct 29 12:02:50 ubuntu kernel: [80713.796695] [ 2514]   121  2514   221662      380   659456     1565             0 gsd-color
Oct 29 12:02:50 ubuntu kernel: [80713.796696] [ 2517]   121  2517    98465        0   442368      487             0 gsd-datetime
Oct 29 12:02:50 ubuntu kernel: [80713.796697] [ 2518]   121  2518    70973        0   172032      168             0 gsd-housekeepin
Oct 29 12:02:50 ubuntu kernel: [80713.796698] [ 2521]   121  2521   160742      668   614400     1881             0 gsd-keyboard
Oct 29 12:02:50 ubuntu kernel: [80713.796699] [ 2523]   121  2523   304013      153   684032     1802             0 gsd-media-keys
Oct 29 12:02:50 ubuntu kernel: [80713.796700] [ 2532]   121  2532    50537        0   159744      148             0 gsd-mouse
Oct 29 12:02:50 ubuntu kernel: [80713.796700] [ 2533]   121  2533   163406      164   626688     1741             0 gsd-power
Oct 29 12:02:50 ubuntu kernel: [80713.796701] [ 2536]   121  2536    66789        0   262144      304             0 gsd-print-notif
Oct 29 12:02:50 ubuntu kernel: [80713.796702] [ 2538]   121  2538    50542        0   151552      147             0 gsd-rfkill
Oct 29 12:02:50 ubuntu kernel: [80713.796703] [ 2542]   121  2542    68971        2   172032      157             0 gsd-screensaver
Oct 29 12:02:50 ubuntu kernel: [80713.796704] [ 2545]   121  2545    76351        0   221184      280             0 gsd-sharing
Oct 29 12:02:50 ubuntu kernel: [80713.796705] [ 2547]   121  2547    94559        0   229376      260             0 gsd-smartcard
Oct 29 12:02:50 ubuntu kernel: [80713.796706] [ 2550]   121  2550    83288        0   266240      280             0 gsd-sound
Oct 29 12:02:50 ubuntu kernel: [80713.796707] [ 2556]   121  2556   143377        3   598016     1962             0 gsd-wacom
Oct 29 12:02:50 ubuntu kernel: [80713.796708] [ 2579]   121  2579    51281        0   163840      179             0 ibus-engine-sim
Oct 29 12:02:50 ubuntu kernel: [80713.796709] [ 2630]   117  2630    81283      435   258048      795             0 colord
Oct 29 12:02:50 ubuntu kernel: [80713.796710] [ 2693]     0  2693   103971      685   311296     1366             0 gdm-session-wor
Oct 29 12:02:50 ubuntu kernel: [80713.796710] [ 2720]  1000  2720    19315      208   200704      234             0 systemd
Oct 29 12:02:50 ubuntu kernel: [80713.796711] [ 2721]  1000  2721    28630       63   262144      675             0 (sd-pam)
Oct 29 12:02:50 ubuntu kernel: [80713.796712] [ 2735]  1000  2735   201194      209   249856      116             0 gnome-keyring-d
Oct 29 12:02:50 ubuntu kernel: [80713.796713] [ 2739]  1000  2739    53062        0   180224      165             0 gdm-x-session
Oct 29 12:02:50 ubuntu kernel: [80713.796714] [ 2741]  1000  2741   189051    38719   806912     1992             0 Xorg
Oct 29 12:02:50 ubuntu kernel: [80713.796715] [ 2757]  1000  2757    12889      480   147456       76             0 dbus-daemon
Oct 29 12:02:50 ubuntu kernel: [80713.796716] [ 2761]  1000  2761   179069      352   458752      390             0 gnome-session-b
Oct 29 12:02:50 ubuntu kernel: [80713.796717] [ 2839]  1000  2839     2988       10    61440       78             0 ssh-agent
Oct 29 12:02:50 ubuntu kernel: [80713.796718] [ 2842]  1000  2842    87328       65   188416      101             0 at-spi-bus-laun
Oct 29 12:02:50 ubuntu kernel: [80713.796719] [ 2847]  1000  2847    12514      104   147456       73             0 dbus-daemon
Oct 29 12:02:50 ubuntu kernel: [80713.796720] [ 2849]  1000  2849    55197       75   208896      106             0 at-spi2-registr
Oct 29 12:02:50 ubuntu kernel: [80713.796721] [ 2867]  1000  2867   945776    45713  1822720    12107             0 gnome-shell
Oct 29 12:02:50 ubuntu kernel: [80713.796722] [ 2874]  1000  2874    73050      157   196608      103             0 gvfsd
Oct 29 12:02:50 ubuntu kernel: [80713.796723] [ 2879]  1000  2879   104029       14   172032      122             0 gvfsd-fuse
Oct 29 12:02:50 ubuntu kernel: [80713.796724] [ 2892]  1000  2892   471330      603   450560      360             0 pulseaudio
Oct 29 12:02:50 ubuntu kernel: [80713.796725] [ 2924]  1000  2924    91178     1180   200704       94             0 ibus-daemon
Oct 29 12:02:50 ubuntu kernel: [80713.796726] [ 2928]  1000  2928    70249        0   167936      201             0 ibus-dconf
Oct 29 12:02:50 ubuntu kernel: [80713.796727] [ 2930]  1000  2930    86158      174   425984      994             0 ibus-x11
Oct 29 12:02:50 ubuntu kernel: [80713.796728] [ 2936]  1000  2936    69732       74   167936       98             0 ibus-portal
Oct 29 12:02:50 ubuntu kernel: [80713.796729] [ 2943]  1000  2943   172445        0   532480      893             0 gnome-shell-cal
Oct 29 12:02:50 ubuntu kernel: [80713.796752] [ 2947]  1000  2947   343123        0   806912     1089             0 evolution-sourc
Oct 29 12:02:50 ubuntu kernel: [80713.796753] [ 2955]  1000  2955   194127      137   688128     1231             0 goa-daemon
Oct 29 12:02:50 ubuntu kernel: [80713.796754] [ 2972]  1000  2972    47033      156   135168       57             0 dconf-service
Oct 29 12:02:50 ubuntu kernel: [80713.796755] [ 2981]  1000  2981    75903        0   221184      265             0 goa-identity-se
Oct 29 12:02:50 ubuntu kernel: [80713.796756] [ 2985]  1000  2985    95174      389   237568      143             0 gvfs-udisks2-vo
Oct 29 12:02:50 ubuntu kernel: [80713.796757] [ 2990]  1000  2990    72224       77   192512      127             0 gvfs-gphoto2-vo
Oct 29 12:02:50 ubuntu kernel: [80713.796758] [ 2994]  1000  2994    68545       54   163840       89             0 gvfs-goa-volume
Oct 29 12:02:50 ubuntu kernel: [80713.796759] [ 2998]  1000  2998    68994       36   176128       81             0 gvfs-mtp-volume
Oct 29 12:02:50 ubuntu kernel: [80713.796760] [ 3003]  1000  3003    94737       54   229376      165             0 gvfs-afc-volume
Oct 29 12:02:50 ubuntu kernel: [80713.796761] [ 3010]  1000  3010   147893      353   503808      988             0 gsd-power
Oct 29 12:02:50 ubuntu kernel: [80713.796762] [ 3013]  1000  3013    87366      133   290816      232             0 gsd-print-notif
Oct 29 12:02:50 ubuntu kernel: [80713.796763] [ 3017]  1000  3017   105873      124   188416       78             0 gsd-rfkill
Oct 29 12:02:50 ubuntu kernel: [80713.796764] [ 3019]  1000  3019    68971       75   155648       70             0 gsd-screensaver
Oct 29 12:02:50 ubuntu kernel: [80713.796765] [ 3020]  1000  3020   113285      282   237568      151             0 gsd-sharing
Oct 29 12:02:50 ubuntu kernel: [80713.796766] [ 3024]  1000  3024   112993       65   229376      164             0 gsd-smartcard
Oct 29 12:02:50 ubuntu kernel: [80713.796767] [ 3038]  1000  3038    83290       75   262144      200             0 gsd-sound
Oct 29 12:02:50 ubuntu kernel: [80713.796767] [ 3039]  1000  3039   123834      235   450560     1081             0 gsd-xsettings
Oct 29 12:02:50 ubuntu kernel: [80713.796768] [ 3044]  1000  3044   107288      187   462848     1048             0 gsd-wacom
Oct 29 12:02:50 ubuntu kernel: [80713.796769] [ 3050]  1000  3050    69614       87   163840       84             0 gsd-a11y-settin
Oct 29 12:02:50 ubuntu kernel: [80713.796770] [ 3052]  1000  3052    86641      832   442368      967             0 gsd-clipboard
Oct 29 12:02:50 ubuntu kernel: [80713.796771] [ 3057]  1000  3057   167216      428   516096      845             0 gsd-color
Oct 29 12:02:50 ubuntu kernel: [80713.796772] [ 3058]  1000  3058   117509       77   450560      423             0 gsd-datetime
Oct 29 12:02:50 ubuntu kernel: [80713.796773] [ 3059]  1000  3059    91209      105   217088      106             0 gsd-housekeepin
Oct 29 12:02:50 ubuntu kernel: [80713.796774] [ 3062]  1000  3062   126815      987   495616     1880             0 gsd-keyboard
Oct 29 12:02:50 ubuntu kernel: [80713.796775] [ 3066]  1000  3066   302853      539   573440      881             0 gsd-media-keys
Oct 29 12:02:50 ubuntu kernel: [80713.796776] [ 3072]  1000  3072    69615       85   159744       85             0 gsd-mouse
Oct 29 12:02:50 ubuntu kernel: [80713.796777] [ 3098]  1000  3098   127227       72   454656      411             0 gsd-printer
Oct 29 12:02:50 ubuntu kernel: [80713.796778] [ 3113]  1000  3113   716763     5058  1732608     5030             0 goldendict
Oct 29 12:02:50 ubuntu kernel: [80713.796779] [ 3124]  1000  3124   574131    11480  1986560     5766             0 mattermost-desk
Oct 29 12:02:50 ubuntu kernel: [80713.796780] [ 3126]  1000  3126   537667     5577  1638400     4296             0 electron-ssr
Oct 29 12:02:50 ubuntu kernel: [80713.796781] [ 3130]  1000  3130   315507    39053  1441792    15191             0 albert
Oct 29 12:02:50 ubuntu kernel: [80713.796782] [ 3136]  1000  3136    67984      186   167936      205             0 gsd-disk-utilit
Oct 29 12:02:50 ubuntu kernel: [80713.796783] [ 3137]  1000  3137   269178     3910   864256     2546             0 nautilus-deskto
Oct 29 12:02:50 ubuntu kernel: [80713.796784] [ 3144]  1000  3144   217851        0   856064     9968             0 evolution-calen
Oct 29 12:02:50 ubuntu kernel: [80713.796785] [ 3147]  1000  3147    94695      541   229376      230             0 gvfsd-trash
Oct 29 12:02:50 ubuntu kernel: [80713.796786] [ 3161]  1000  3161    86226      166   589824      634             0 mattermost-desk
Oct 29 12:02:50 ubuntu kernel: [80713.796786] [ 3162]  1000  3162    86227        0   593920      800             0 electron-ssr
Oct 29 12:02:50 ubuntu kernel: [80713.796787] [ 3167]  1000  3167    51349      195   159744       81             0 ibus-engine-sim
Oct 29 12:02:50 ubuntu kernel: [80713.796788] [ 3185]  1000  3185   241877        0   647168     9800             0 evolution-calen
Oct 29 12:02:50 ubuntu kernel: [80713.796789] [ 3229]  1000  3229    20053       17   192512      114             0 gconfd-2
Oct 29 12:02:50 ubuntu kernel: [80713.796790] [ 3249]  1000  3249   115069     2697   786432     2011             0 mattermost-desk
Oct 29 12:02:50 ubuntu kernel: [80713.796791] [ 3260]  1000  3260   271042     5867  1486848     4755           300 mattermost-desk
Oct 29 12:02:50 ubuntu kernel: [80713.796792] [ 3292]  1000  3292    51376       12   159744      338             0 gvfsd-metadata
Oct 29 12:02:50 ubuntu kernel: [80713.796793] [ 3314]  1000  3314   112882      203   782336     2254             0 electron-ssr
Oct 29 12:02:50 ubuntu kernel: [80713.796794] [ 3324]  1000  3324   277671       77  1785856    12706           300 electron-ssr
Oct 29 12:02:50 ubuntu kernel: [80713.796795] [ 3358]  1000  3358    14395      779   147456     1088             0 python
Oct 29 12:02:50 ubuntu kernel: [80713.796796] [ 3497]  1000  3497   387165     4587  1638400     4631             0 gnome-control-c
Oct 29 12:02:50 ubuntu kernel: [80713.796797] [ 3526]  1000  3526    87963      383   262144      149             0 ibus-engine-rim
Oct 29 12:02:50 ubuntu kernel: [80713.796798] [ 3551]  1000  3551   182499     3073   712704     1449             0 gnome-terminal-
Oct 29 12:02:50 ubuntu kernel: [80713.796799] [ 3558]  1000  3558    14499      168   151552     1161             0 zsh
Oct 29 12:02:50 ubuntu kernel: [80713.796800] [ 3606]  1000  3606    35905    26683   315392      140             0 tmux: server
Oct 29 12:02:50 ubuntu kernel: [80713.796801] [ 3638]  1000  3638    15159     1311   155648      173             0 zsh
Oct 29 12:02:50 ubuntu kernel: [80713.796802] [ 3785]  1000  3785   355174     5946  1085440    14987             0 gnome-software
Oct 29 12:02:50 ubuntu kernel: [80713.796803] [ 3793]  1000  3793   171641      904   647168     1230             0 update-notifier
Oct 29 12:02:50 ubuntu kernel: [80713.796804] [ 3805]     0  3805   138983      749   499712     1202             0 fwupd
Oct 29 12:02:50 ubuntu kernel: [80713.796805] [ 3908]  1000  3908    14215        9   147456      524             0 zsh
Oct 29 12:02:50 ubuntu kernel: [80713.796806] [ 4270]  1000  4270    14730      726   155648      344             0 zsh
Oct 29 12:02:50 ubuntu kernel: [80713.796807] [ 4365]  1000  4365   197460      315   724992     1073             0 deja-dup-monito
Oct 29 12:02:50 ubuntu kernel: [80713.796808] [13924]     0 13924    19794      141   200704        0             0 sudo
Oct 29 12:02:50 ubuntu kernel: [80713.796809] [13925]     0 13925     7571      506    98304        1             0 bash
Oct 29 12:02:50 ubuntu kernel: [80713.796810] [14280]  1000 14280     2988       68    77824        0             0 ssh-agent
Oct 29 12:02:50 ubuntu kernel: [80713.796811] [14285]  1000 14285    14055      207   126976        0             0 ssh
Oct 29 12:02:50 ubuntu kernel: [80713.796812] [14384]  1000 14384    15126     1461   155648        0             0 zsh
Oct 29 12:02:50 ubuntu kernel: [80713.796813] [14467]  1000 14467    14055      235   135168        0             0 ssh
Oct 29 12:02:50 ubuntu kernel: [80713.796814] [14693]  1000 14693   962275    71745  3153920        0             0 firefox
Oct 29 12:02:50 ubuntu kernel: [80713.796815] [14812]  1000 14812  5480555    37535  2330624        0             0 WebExtensions
Oct 29 12:02:50 ubuntu kernel: [80713.796816] [14882]  1000 14882   100841     1760   593920        0             0 RDD Process
Oct 29 12:02:50 ubuntu kernel: [80713.796817] [14893]  1000 14893   724960    19040  1765376        0             0 Web Content
Oct 29 12:02:50 ubuntu kernel: [80713.796818] [23228]  1000 23228   131675      387   229376        0             0 gvfsd-network
Oct 29 12:02:50 ubuntu kernel: [80713.796819] [23235]  1000 23235   193682      745   720896        0             0 gvfsd-smb-brows
Oct 29 12:02:50 ubuntu kernel: [80713.796820] [23242]  1000 23242    95357      266   233472        0             0 gvfsd-dnssd
Oct 29 12:02:50 ubuntu kernel: [80713.796821] [23322]  1000 23322   108449      338   204800        0             0 zeitgeist-daemo
Oct 29 12:02:50 ubuntu kernel: [80713.796822] [23330]  1000 23330    81685      633   229376        0             0 zeitgeist-fts
Oct 29 12:02:50 ubuntu kernel: [80713.796823] [24469]  1000 24469   249243    13972  1572864        0             0 chrome
Oct 29 12:02:50 ubuntu kernel: [80713.796824] [24474]  1000 24474     3679       16    65536        0             0 cat
Oct 29 12:02:50 ubuntu kernel: [80713.796825] [24475]  1000 24475     3679       16    69632        0             0 cat
Oct 29 12:02:50 ubuntu kernel: [80713.796826] [24479]  1000 24479   110703     2571   745472        0             0 chrome
Oct 29 12:02:50 ubuntu kernel: [80713.796828] [24480]  1000 24480     6716      143    98304        0             0 nacl_helper
Oct 29 12:02:50 ubuntu kernel: [80713.796829] [24483]  1000 24483   110703     2575   565248        0             0 chrome
Oct 29 12:02:50 ubuntu kernel: [80713.796830] [24503]  1000 24503   136354     5844   901120        0           200 chrome
Oct 29 12:02:50 ubuntu kernel: [80713.796831] [24508]  1000 24508   128286     4106   811008        0           200 chrome
Oct 29 12:02:50 ubuntu kernel: [80713.796832] [24674]  1000 24674   192377    13715  1806336        0           300 chrome
Oct 29 12:02:50 ubuntu kernel: [80713.796833] [24915]  1000 24915   193359    11080  1728512        0           300 chrome
Oct 29 12:02:50 ubuntu kernel: [80713.796834] [25050]     0 25050     3653       15    61440        0             0 tail
Oct 29 12:02:50 ubuntu kernel: [80713.796835] [25952]     0 25952     5511      180    77824        9             0 mount.ntfs
Oct 29 12:02:50 ubuntu kernel: [80713.796836] [25976]     0 25976     5754      460    73728        0             0 mount.ntfs
Oct 29 12:02:50 ubuntu kernel: [80713.796837] [29499]  1000 29499    14055      209   126976        0             0 ssh
Oct 29 12:02:50 ubuntu kernel: [80713.796838] [30274]  1000 30274    89943      320   266240        0             0 gvfsd-http
Oct 29 12:02:50 ubuntu kernel: [80713.796839] [32378]  1000 32378   167237     3410  1028096        0           300 chrome
Oct 29 12:02:50 ubuntu kernel: [80713.796840] [  730]     0   730    34466      471   245760        0             0 nginx
Oct 29 12:02:50 ubuntu kernel: [80713.796841] [  803]  1000   803    14272      607   151552        0             0 zsh
Oct 29 12:02:50 ubuntu kernel: [80713.796842] [28780]  1000 28780   340185     5495   921600        0             0 nautilus
Oct 29 12:02:50 ubuntu kernel: [80713.796843] [ 1212]  1000  1212   372443     6269  1277952        0             0 VirtualBox
Oct 29 12:02:50 ubuntu kernel: [80713.796844] [ 1233]  1000  1233    46863     1023   315392        0             0 VBoxXPCOMIPCD
Oct 29 12:02:50 ubuntu kernel: [80713.796845] [ 1238]  1000  1238   229668     6095   602112        0             0 VBoxSVC
Oct 29 12:02:50 ubuntu kernel: [80713.796845] [ 1458]  1000  1458  1737236  1334969 11362304        0             0 VBoxHeadless
Oct 29 12:02:50 ubuntu kernel: [80713.796846] [ 1519]  1000  1519   402033     9283  1249280        0             0 VirtualBox
Oct 29 12:02:50 ubuntu kernel: [80713.796847] [ 1965]  1000  1965    14055      209   131072        0             0 ssh
Oct 29 12:02:50 ubuntu kernel: [80713.796848] [ 3710]  1000  3710    14055      240   118784        0             0 ssh
Oct 29 12:02:50 ubuntu kernel: [80713.796849] [ 3968]  1000  3968     4998       65    77824        0             0 wps
Oct 29 12:02:50 ubuntu kernel: [80713.796850] [ 3971]  1000  3971   904805    43542  1966080        0             0 wps
Oct 29 12:02:50 ubuntu kernel: [80713.796851] [11126]  1000 11126   776309    36439  2531328        0             0 Web Content
Oct 29 12:02:50 ubuntu kernel: [80713.796852] [11263]  1000 11263   844950    37587  2871296        0             0 Web Content
Oct 29 12:02:50 ubuntu kernel: [80713.796853] [17639]  1000 17639    15118      503   147456        0             0 mosh-server
Oct 29 12:02:50 ubuntu kernel: [80713.796854] [17640]  1000 17640    14203      529   143360        0             0 zsh
Oct 29 12:02:50 ubuntu kernel: [80713.796855] [27028]  1000 27028     7083       88    94208        0             0 tmux: client
Oct 29 12:02:50 ubuntu kernel: [80713.796856] [27176]  1000 27176    14060      265   135168        0             0 ssh
Oct 29 12:02:50 ubuntu kernel: [80713.796857] [27454]  1000 27454    14049      211   126976        0             0 ssh
Oct 29 12:02:50 ubuntu kernel: [80713.796858] [29359]  1000 29359    14047      208   135168        0             0 ssh
Oct 29 12:02:50 ubuntu kernel: [80713.796859] [ 3677]     0  3677    27053      116   245760      327             0 cupsd
Oct 29 12:02:50 ubuntu kernel: [80713.796860] [ 3679]     0  3679    75881      127   339968      254             0 cups-browsed
Oct 29 12:02:50 ubuntu kernel: [80713.796861] [ 3683]     7  3683    21603      155   192512       48             0 dbus
Oct 29 12:02:50 ubuntu kernel: [80713.796862] [24911]  1000 24911   776312    42262  2584576        0             0 Web Content
Oct 29 12:02:50 ubuntu kernel: [80713.796863] [25337]     0 25337    26837      294    73728        0          -999 containerd-shim
Oct 29 12:02:50 ubuntu kernel: [80713.796864] [25358]     0 25358     1482        5    36864       46             0 entrypoint.sh
Oct 29 12:02:50 ubuntu kernel: [80713.796865] [25449]     0 25449     1693        0    40960       93             0 su
Oct 29 12:02:50 ubuntu kernel: [80713.796866] [25451]  1000 25451     1411        0    40960       37             0 sh
Oct 29 12:02:50 ubuntu kernel: [80713.796867] [25452]  1000 25452     1411        0    36864       37             0 run.sh
Oct 29 12:02:50 ubuntu kernel: [80713.796868] [25453]  1000 25453     1444        3    36864       94             0 run.sh
Oct 29 12:02:50 ubuntu kernel: [80713.796869] [25645]  1000 25645     1411        0    40960       36             0 deepin-wine
Oct 29 12:02:50 ubuntu kernel: [80713.796870] [25647]  1000 25647   686359    29338   679936     4630             0 WeChat.exe
Oct 29 12:02:50 ubuntu kernel: [80713.796871] [25652]  1000 25652     2154      968    45056       22             0 wineserver.real
Oct 29 12:02:50 ubuntu kernel: [80713.796872] [25664]  1000 25664   663145        2   118784      189             0 services.exe
Oct 29 12:02:50 ubuntu kernel: [80713.796873] [25669]  1000 25669   663819        2   114688      201             0 winedevice.exe
Oct 29 12:02:50 ubuntu kernel: [80713.796874] [25681]  1000 25681   662580        2    98304      160             0 plugplay.exe
Oct 29 12:02:50 ubuntu kernel: [80713.796875] [25689]  1000 25689   667635       45   147456      426             0 winedevice.exe
Oct 29 12:02:50 ubuntu kernel: [80713.796875] [25728]  1000 25728   674023     1198   163840        6             0 explorer.exe
Oct 29 12:02:50 ubuntu kernel: [80713.796876] [25799]  1000 25799   161828     3744   954368        0           300 chrome
Oct 29 12:02:50 ubuntu kernel: [80713.796877] [26024]  1000 26024   697196    12862  1343488        0             0 Web Content
Oct 29 12:02:50 ubuntu kernel: [80713.796878] [26062]  1000 26062   795448    41424  2347008        0             0 Web Content
Oct 29 12:02:50 ubuntu kernel: [80713.796879] [27615]  1000 27615   647124     4749   843776        0             0 Web Content
Oct 29 12:02:50 ubuntu kernel: [80713.796880] [28597]  1000 28597    13974      176   139264        0             0 ssh
Oct 29 12:02:50 ubuntu kernel: [80713.796881] [29267]     0 29267  1826854  1583249 14639104        0             0 nginx
Oct 29 12:02:50 ubuntu kernel: [80713.796882] [29670]  1000 29670    12165      116   131072        0             0 top
Oct 29 12:02:50 ubuntu kernel: [80713.796883] [29833]     0 29833     3653       16    73728        0             0 tail
Oct 29 12:02:50 ubuntu kernel: [80713.796885] Out of memory: Kill process 29267 (nginx) score 355 or sacrifice child
Oct 29 12:02:50 ubuntu kernel: [80713.796888] Killed process 29267 (nginx) total-vm:7307416kB, anon-rss:6328164kB, file-rss:764kB, shmem-rss:4068kB
Oct 29 12:02:50 ubuntu kernel: [80714.106488] oom_reaper: reaped process 29267 (nginx), now anon-rss:0kB, file-rss:0kB, shmem-rss:4068kB
Oct 29 12:02:51 ubuntu kernel: [80715.347890] net_ratelimit: 43582 callbacks suppressed
Oct 29 12:02:51 ubuntu kernel: [80715.347890] TCP: too many orphaned sockets

comment:12 in reply to: ↑ 10 Changed 3 weeks ago by mdounin

  1. Why RES gradually increase

As explained in comment:9, this is because RES reflects memory actually touched by the process. For example, if you allocate a 2M buffer and only use 1M for actual data, RES will be 1M. If later you'll reuse this buffer for 1.5M of actual data, RES will grow to 1.5M.

In case of nginx, subsequent runs are likely to touch slightly different regions of memory, resulting in RES asymptotically growing (in simple cases it will grow till it reaches VIRT, which reflects total memory address space used by the process).

  1. If RES does not reflect the memory leak, how to determine if there is a memory leak

Usual approach is to check if the memory usage grows without bound under constant load. As long as "without bound" is true, it doesn't matter if you'll use RES or VIRT, but you may find VIRT to be more informative when looking for memory leaks.

Note: See TracTickets for help on using tickets.