High memory consumed by Kong for secure websocket

The memory usage of Kong Pod is abnormally high when handling secure WebSockets. We have deployed Kong in the Azure AKS cluster to handle HTTP and WebSocket connections using the Azure L4 load balancer. The TLS termination is handled by Kong. For HTTPS requests, Kong’s memory usage is normal. However, for secure WebSockets, the memory usage is exceptionally high:

  • When there are no WebSocket connections, Kong uses 250MB of memory, this is normal.
  • With 100 concurrent WebSocket connections(100 CCU), Kong consumes about 800MB of memory.
  • With 200 CCU, Kong consumes 1.6GB of memory.
  • When all WebSocket connections are closed, the memory quickly returns to normal.

This is the case in both Kong v3.2.2 and v3.7.0. Is this related to how TLS termination is handled in Kong?

These are some environment variables configured in the Kong pod:

  ...
  containers:
  - env:
    - name: KONG_CLIENT_BODY_BUFFER_SIZE
      value: 32k
    - name: KONG_CLIENT_MAX_BODY_SIZE
      value: "0"
    - name: KONG_DNS_ORDER
      value: A
    - name: KONG_NGINX_HTTP_CLIENT_BODY_BUFFER_SIZE
      value: 32k
    - name: KONG_NGINX_WORKER_PROCESSES
      value: "1"
    - name: KONG_PROXY_ACCESS_LOG
      value: "off"
    - name: KONG_PROXY_LISTEN
      value: 0.0.0.0:8000, 0.0.0.0:8443 ssl
    - name: KONG_STATUS_LISTEN
      value: 0.0.0.0:8100
    - name: KONG_NGINX_HTTP_LUA_SHARED_DICT
      value: fleet_custom_data 16k
    - name: KONG_NGINX_PROXY_PROXY_BUFFERING
      value: "on"
    - name: KONG_NGINX_PROXY_PROXY_REQUEST_BUFFERING
      value: "on"
    - name: KONG_NGINX_PROXY_PROXY_BUFFER_SIZE
      value: 8k
    - name: KONG_NGINX_PROXY_PROXY_BUFFERS
      value: 128 4k
    - name: KONG_NGINX_PROXY_PROXY_BUSY_BUFFERS_SIZE
      value: 32k
    - name: KONG_NGINX_PROXY_KEEPALIVE_TIMEOUT
      value: "620"
    - name: KONG_NGINX_PROXY_KEEPALIVE_REQUESTS
      value: "10000"
    - name: KONG_DATABASE
      value: "off"
    - name: KONG_ROLE
      value: data_plane
    - name: KONG_CLUSTER_MTLS
      value: pki
    - name: KONG_CLUSTER_CA_CERT
      value: /tls/ca.crt
    - name: KONG_CLUSTER_CERT
      value: /tls/tls.crt
    - name: KONG_CLUSTER_CERT_KEY
      value: /tls/tls.key
    - name: KONG_LUA_SSL_TRUSTED_CERTIFICATE
      value: /tls/ca.crt
    - name: KONG_CLUSTER_SERVER_NAME
      value: kong_clustering
    - name: KONG_SSL_CERT
      value: /kong-tls/tls.crt
    - name: KONG_SSL_CERT_KEY
      value: /kong-tls/tls.key
    - name: KONG_SSL_CIPHER_SUITE
      value: custom
    - name: KONG_SSL_CIPHERS
      value: TLS_AES_128_GCM_SHA256
    - name: WORKER_CONNECTIONS
      value: "16384"
    - name: NGINX_MAIN_WORKER_RLIMIT_NOFILE
      value: "32768"