Inconsistent Load Balancing Behavior When Using Kong in Kubernetes Cluster

Kubernetes
1

Hello Kong Community,

I’m experiencing an issue with load balancing in my Kubernetes environment where Kong is being used as an API gateway. While direct curl requests to the Kubernetes service are properly load balanced in a round-robin fashion, requests routed through Kong are not being evenly distributed among pods.

Environment:

  • Kubernetes version: v1.25.11-eks-a5565ad
  • Kong version: 3.1.1
  • Nginx version: openresty/1.21.4.1
  • Deployment strategy: kong is deployed as a NodePort Service.

Issue Description:

  • When making direct requests to the Kubernetes service (e.g., curl http://service-name.namespace.svc.cluster.local), the load is evenly distributed among all available pods, indicating that kube-proxy’s round-robin configuration is functioning correctly.
  • However, when requests are routed through Kong (e.g., curl -H "Host: example.com" http://<node-ip>:<node-port>), they are not evenly distributed. Instead, requests seem to be directed towards a single pod for a certain duration before switching to another pod.

Troubleshooting Steps Taken:

  • Checked the Kong upstream configuration, which is set to round-robin.
  • Inspected the nginx-kong.conf to ensure no custom load balancing settings are applied.
  • Increased the log level of Kong and reviewed the logs, but did not find any obvious issues related to load balancing decisions.

Questions:

  1. Could there be a configuration within Kong or its underlying Nginx setup that might be causing this uneven load distribution?
  2. Are there known issues or behaviors in Kong that could lead to such a scenario, particularly when integrated within a Kubernetes cluster?
  3. Any suggestions on further steps to diagnose or resolve this issue would be greatly appreciated.

I have attached relevant configuration files for reference. Any help or guidance on this matter would be immensely helpful.

Thank you in advance for your assistance!

Best regards,
takahashi-jo

kong

SERVICE
{
“connect_timeout”: 60000,
“read_timeout”: 60000,
“name”: “namespace.service-name.service-name.80”,
“tls_verify”: null,
“tls_verify_depth”: null,
“host”: “service-name.namespace.80.svc”,
“created_at”: 1705654718,
“updated_at”: 1705654718,
“tags”: [
“managed-by-ingress-controller”
],
“ca_certificates”: null,
“id”: “92a44563-bff4-42f5-8fba-2c72d8be8360”,
“protocol”: “http”,
“enabled”: true,
“port”: 80,
“path”: “/”,
“client_certificate”: null,
“write_timeout”: 60000,
“retries”: 5
}

ROUTE
{
“hosts”: [
“example.com”
],
“path_handling”: “v0”,
“service”: {
“id”: “92a44563-bff4-42f5-8fba-2c72d8be8360”
},
“name”: “namespace.service-name.service-name.example.com.80”,
“strip_path”: true,
“snis”: null,
“created_at”: 1705654877,
“updated_at”: 1705654877,
“request_buffering”: true,
“paths”: [
“/api/example”
],
“methods”: null,
“sources”: null,
“id”: “33af07d8-e844-4d19-b7e0-2fdec963a236”,
“preserve_host”: true,
“tags”: [
“managed-by-ingress-controller”
],
“headers”: null,
“destinations”: null,
“protocols”: [
“http”,
“https”
],
“https_redirect_status_code”: 426,
“regex_priority”: 0,
“response_buffering”: true
}

UPSTREAM
{
“hash_on_query_arg”: null,
“hash_on_uri_capture”: null,
“hash_on_cookie”: null,
“hash_on_cookie_path”: “/”,
“hash_fallback”: “none”,
“slots”: 10000,
“algorithm”: “round-robin”,
“hash_fallback_uri_capture”: null,
“client_certificate”: null,
“host_header”: null,
“id”: “ab44cadd-697a-45d5-bc09-d94be3ba18f5”,
“created_at”: 1705651412,
“tags”: [
“managed-by-ingress-controller”
],
“hash_fallback_header”: null,
“healthchecks”: {
“active”: {
“type”: “http”,
“concurrency”: 10,
“healthy”: {
“interval”: 0,
“successes”: 0,
“http_statuses”: [
200,
302
]
},
“headers”: null,
“http_path”: “/”,
“https_sni”: null,
“https_verify_certificate”: true,
“timeout”: 1,
“unhealthy”: {
“http_statuses”: [
429,
404,
500,
501,
502,
503,
504,
505
],
“tcp_failures”: 0,
“timeouts”: 0,
“interval”: 0,
“http_failures”: 0
}
},
“passive”: {
“healthy”: {
“successes”: 0,
“http_statuses”: [
200,
201,
202,
203,
204,
205,
206,
207,
208,
226,
300,
301,
302,
303,
304,
305,
306,
307,
308
]
},
“unhealthy”: {
“http_statuses”: [
429,
500,
503
],
“tcp_failures”: 0,
“timeouts”: 0,
“http_failures”: 0
},
“type”: “http”
},
“threshold”: 0
},
“hash_on”: “none”,
“name”: “service-name.namespace.80.svc”,
“use_srv_name”: false,
“hash_on_header”: null,
“hash_fallback_query_arg”: null
}

kubernetes

kube-proxy config
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
acceptContentTypes: “”
burst: 10
contentType: application/vnd.kubernetes.protobuf
kubeconfig: /var/lib/kube-proxy/kubeconfig
qps: 5
clusterCIDR: “”
configSyncPeriod: 15m0s
conntrack:
maxPerCore: 32768
min: 131072
tcpCloseWaitTimeout: 1h0m0s
tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: “”
iptables:
masqueradeAll: false
masqueradeBit: 14
minSyncPeriod: 0s
syncPeriod: 30s
ipvs:
excludeCIDRs: null
minSyncPeriod: 0s
scheduler: “rr”
syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: “ipvs”
nodePortAddresses: null
oomScoreAdj: -998
portRange: “”
udpIdleTimeout: 250ms

Service
apiVersion: v1
kind: Service
metadata:
annotations:
…
creationTimestamp: “2024-01-19T04:45:40Z”
labels:
…
name: service-name
namespace: namespace
resourceVersion: “152616687”
uid: 2024579a-74ab-4a29-8bac-49c4d387d397
spec:
clusterIP: 172.20.253.250
clusterIPs:

  • 172.20.253.250
    internalTrafficPolicy: Cluster
    ipFamilies:
  • IPv4
    ipFamilyPolicy: SingleStack
    ports:
  • name: http
    port: 80
    protocol: TCP
    targetPort: 80
    selector:
    app: service-name
    version: v1
    sessionAffinity: None
    type: ClusterIP
    status:
    loadBalancer: {}