Kong memory usage doesn't come down

1

We performed some load tests with Kong 1.2.0 on Openshift 3.11.

The pod spec has memory limits of 1Gi to 4G.

Kong-memory
Kong-memory.PNG3253×1420 142 KB

During the tests, the memory went up to 3Gi. But post the tests, the memory consumption still shows 3Gi and it never gets released.

Attached image for reference. The CPU utilization fluctuates, but the memory never comes down.

Below is the snapshot long after the tests are run and no requests are being processed while taking this snapshot. Why are there 16 nginx processes still running?

/ # top -bn1                                                                                                                                    
Mem: 64840936K used, 847196K free, 60152K shrd, 0K buff, 27034820K cached                                                                       
CPU:  12% usr  16% sys   0% nic  70% idle   0% io   0% irq   0% sirq                                                                            
Load average: 7.46 6.64 6.65 7/3663 106                                                                                                         
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND                                                                                           
   35     1 nobody   S    3019m   4%   8   0% nginx: worker process                                                                             
   36     1 nobody   S    2753m   4%   8   0% nginx: worker process                                                                             
   37     1 nobody   S    2435m   4%   8   0% nginx: worker process                                                                             
   42     1 nobody   S    2337m   3%   4   0% nginx: worker process                                                                             
   47     1 nobody   S    2299m   3%   0   0% nginx: worker process                                                                             
   41     1 nobody   S    2275m   3%   6   0% nginx: worker process                                                                             
   40     1 nobody   S    2263m   3%  14   0% nginx: worker process                                                                             
   43     1 nobody   S    2248m   3%   7   0% nginx: worker process                                                                             
   45     1 nobody   S    2245m   3%   4   0% nginx: worker process                                                                             
   48     1 nobody   S    2234m   3%   0   0% nginx: worker process                                                                             
   46     1 nobody   S    2234m   3%   6   0% nginx: worker process                                                                             
   49     1 nobody   S    2231m   3%   4   0% nginx: worker process                                                                             
   50     1 nobody   S    2227m   3%   8   0% nginx: worker process                                                                             
   44     1 nobody   S    2224m   3%   6   0% nginx: worker process                                                                             
   52     1 nobody   S    2222m   3%  14   0% nginx: worker process                                                                             
   51     1 nobody   S    2219m   3%   4   0% nginx: worker process                                                                             
    1     0 root     S    2143m   3%  13   0% nginx: master process /usr/local/openresty/nginx/sbin/nginx -p /usr/local/kong -c nginx.conf      
   67    61 root     S     1540   0%   3   0% /bin/sh                                                                                           
   95    89 root     S     1540   0%  15   0% /bin/sh                                                                                           
   59    53 root     S     1540   0%   9   0% /bin/sh                                                                                           
   88    81 root     S     1540   0%   1   0% /bin/sh                                                                                           
   81     0 root     S     1528   0%   6   0% /bin/sh -i -c TERM=xterm /bin/sh                                                                  
   53     0 root     S     1528   0%   9   0% /bin/sh -i -c TERM=xterm /bin/sh                                                                  
   61     0 root     S     1528   0%   5   0% /bin/sh -i -c TERM=xterm /bin/sh                                                                  
   89     0 root     S     1528   0%   7   0% /bin/sh -c TERM="cygwin" /bin/sh                                                                  
  106    95 root     R     1528   0%  13   0% top -bn1
2

Are you using Kong Ingress controller?
If yes, please upgrade to 1.2.1, which contains a memory leak fix.

3

No, we are NOT using KIC. Just tried with 1.2.1 and its same.

Kong-memory-1_2_1
Kong-memory-1_2_1.PNG1589×1232 55.9 KB

4

The problem here is that Kong is running as a container on a host with ~16 cores, meaning there are 16 worker process running, each taking up some memory.

This is due to the nginx_worker_processes configuration value, which defaults to auto, meaning it will spin up as many worker process as present on the underlying hardware (or the Virtual Machine).

You should consider setting NGINX_WORKER_PROCESS to 2 or 4.

1 Like
5

Hello @cedar715,

This is more or less expected behavior in regard to how LuaJIT VM is used by Kong. As long as the memory does not grow unconfined it is not considered as a memory leak.

The reason behind it is that Kong heavily caches various objects inside the Lua VM for speeding up the access, which does not get freed when the load is gone. Plus the LuaJIT VM itself does not always gives back freed memory to the O/S for various reasons (fragmentation, avoid frequent system calls, etc).

If you are running low on RAM, scaling down the number of Nginx workers Kong uses with nginx_worker_processes will help.

BTW: the VSZ shown does not reflect the amount of physical memory Kong uses. RSS is more closer to that you may be looking for.

3 Likes