Kong Ingress Controller missing endpoint visibility

Questions
,
1

Hi all,

I’m trying to installa KIC 3.5 o a AKS cluster and I use the chart kong/kong to create 2 separated deployment. The first deployment is used to deploy only the kong gateway/proxy and configure Azure load balancer. The endpoints are correctly present behind the healess service:

kubectl get endpoints -n ingress ingress-kong-proxy-kong-admin
Warning: v1 Endpoints is deprecated in v1.33+; use ``discovery.k8s.io/v1`` EndpointSlice
NAME ENDPOINTS AGE
ingress-kong-proxy-kong-admin 10.244.4.172:8001,10.244.5.107:8001 103m

The second deployment is used to deploy the ingress controller part and the pod fails the startup with this issue:

2025-09-26T17:20:07Z error setup.initialize-kong-clients Failed to create kong client(s), retrying… {“delay”: 1, “error”: “no endpoints for service: "ingress/ingress-kong-proxy-kong-admin"”}
2025-09-26T17:20:08Z error setup.initialize-kong-clients Failed to create kong client(s), retrying… {“delay”: 1, “error”: “no endpoints for service: "ingress/ingress-kong-proxy-kong-admin"”}
2025-09-26T17:20:09Z error setup.initialize-kong-clients Failed to create kong client(s), retrying… {“delay”: 1, “error”: “no endpoints for service: "ingress/ingress-kong-proxy-kong-admin"”}

I suspect that the serviceaccount created from the helm chart hasn’t the correct permissions but I also configured the RBAC under the ingressController node like:

rbac:create: trueenableClusterRoles: truegatewayAPI:enabled: truerules:- apiGroups: [“”]resources: [“services”, “endpoints”, “secrets”, “configmaps”]verbs: [“get”, “list”, “watch”]- apiGroups: [“”]resources: [“services/status”]verbs: [“update”]- apiGroups: [“”]resources: [“events”]verbs: [“create”, “patch”]- apiGroups: [“gateway.networking.k8s.io”]resources: [“gatewayclasses”, “gateways”, “httproutes”, “grpcroutes”, “tcproutes”, “tlsroutes”, “udproutes”, “referencegrants”, “backendtlspolicies”]verbs: [“get”, “list”, “watch”]- apiGroups: [“gateway.networking.k8s.io”]resources: [“gatewayclasses/status”, “gateways/status”, “httproutes/status”, “grpcroutes/status”, “tcproutes/status”, “tlsroutes/status”, “udproutes/status”]verbs: [“update”]- apiGroups: [“configuration.konghq.com”]resources: [““]verbs: [””]- apiGroups: [“networking.k8s.io”]resources: [“ingresses”, “ingressclasses”]verbs: [“get”, “list”, “watch”]- apiGroups: [“networking.k8s.io”]resources: [“ingresses/status”]verbs: [“update”]- apiGroups: [“”]resources: [“pods”, “nodes”]verbs: [“get”, “list”, “watch”]- apiGroups: [“admissionregistration.k8s.io”]resources: [“validatingadmissionconfigurations”]verbs: [“get”, “list”, “watch”, “create”, “update”, “patch”, “delete”]

Could someone help me to identify the root cause of this issue?

Thanks,

Marcello

2

I’m trying to debug deeper the issue and I installed KIC 3.5 in sidecar mode and it worked fine. I also tried to install KIC in 2 different deployment and disable the gateway discovery on the controller and point to the admin service via service dns and also in this configuration worked fine. I cannot use the service DNS because the configuration update from the controller will be deployer on a single gateway pod.
I noticed that the serviceaccount used on the controller didn’t have the permission to access only to the endpointslices and not endpoints and I tried to force the permissions with these manifests:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kong-controller-endpoints-access
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kong-controller-endpoints-access
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kong-controller-endpoints-access
subjects:
- kind: ServiceAccount
  name: ingress-kong-controller-kong
  namespace: ingress

I had no luck with this change.

Marcello

3

I also tried to apply the parameter publishNotReadyAddresses as reported on Kong/kubernetes-ingress-controller#5110 and change the CONTROLLER_GATEWAY_DISCOVERY_DNS_STRATEGY to"pod" without successo.

I tried the deployment with versions 3.1, 3.2, 3.3 and 3.4 without success.

I tested the serviceaccount from the kong proxy and the endpoints and endpointslices reported ready gateway/proxy pods:

# kubectl proxy --port=8001 &
# curl -s "http://localhost:8001/api/v1/namespaces/ingress/endpoints/ingress-kong-proxy-kong-admin" | jq '.subsets[].addresses'
[
  {
    "ip": "10.244.4.244",
    "nodeName": "aks-infra-24197528-vmss000001",
    "targetRef": {
      "kind": "Pod",
      "namespace": "ingress",
      "name": "ingress-kong-proxy-kong-f9vr2",
      "uid": "c15f05a0-acf6-40b9-a80e-a7b2a6f14fb0"
    }
  },
  {
    "ip": "10.244.5.43",
    "nodeName": "aks-infra-24197528-vmss000000",
    "targetRef": {
      "kind": "Pod",
      "namespace": "ingress",
      "name": "ingress-kong-proxy-kong-xzch6",
      "uid": "9724ab7c-d015-412a-8dc2-29c55d15286a"
    }
  }
# curl -s "http://localhost:8001/apis/discovery.k8s.io/v1/namespaces/ingress/endpointslices?labelSelector=kubernetes.io%2Fservice-name%3Dingress-kong-proxy-kong-admin" | jq '.items[].endpoints'
[
  {
    "addresses": [
      "10.244.5.43"
    ],
    "conditions": {
      "ready": true,
      "serving": true,
      "terminating": false
    },
    "targetRef": {
      "kind": "Pod",
      "namespace": "ingress",
      "name": "ingress-kong-proxy-kong-xzch6",
      "uid": "9724ab7c-d015-412a-8dc2-29c55d15286a"
    },
    "nodeName": "aks-infra-24197528-vmss000000",
    "zone": "westeurope-2"
  },
  {
    "addresses": [
      "10.244.4.244"
    ],
    "conditions": {
      "ready": true,
      "serving": true,
      "terminating": false
    },
    "targetRef": {
      "kind": "Pod",
      "namespace": "ingress",
      "name": "ingress-kong-proxy-kong-f9vr2",
      "uid": "c15f05a0-acf6-40b9-a80e-a7b2a6f14fb0"
    },
    "nodeName": "aks-infra-24197528-vmss000001",
    "zone": "westeurope-3"
  }
]
# curl -s "http://localhost:8001/api/v1/namespaces/ingress/endpoints/ingress-kong-proxy-kong-admin" | jq '.subsets[].addresses | length'
2
# curl -s "http://localhost:8001/apis/discovery.k8s.io/v1/namespaces/ingress/endpointslices?labelSelector=kubernetes.io%2Fservice-name%3Dingress-kong-proxy-kong-admin" | jq '.items[].endpoints | length'
2

Marcello

4
5

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.