Subject
DB-less Kong dataplane intermittently remains with empty configuration (configuration_hash=000000…) after rollout until delayed reconciliation or KIC restart
Severity
Medium to High
During Kong proxy rollouts, newly created dataplane pods can remain NotReady for extended periods (10-20+ minutes observed), reducing available gateway capacity and increasing deployment risk.
Kubernetes
Client Version: v1.34.4
Server Version: v1.34.9-gke.1655001
Kong Gateway
Issue reproduced on:
- kong:3.9.0
- kong:3.9.3
Kong Ingress Controller (KIC)
Issue reproduced on:
- kong/kubernetes-ingress-controller:3.5.7
- kong/kubernetes-ingress-controller:3.5.13
Deployment Topology
DB-less Kong deployment managed by KIC.
Controller configuration:
CONTROLLER_KONG_ADMIN_URL=https://kong-admin:8444
CONTROLLER_DUMP_CONFIG=true
CONTROLLER_KONG_ADMIN_TLS_SKIP_VERIFY=true
CONTROLLER_PUBLISH_SERVICE=
Observed controller image:
kong/kubernetes-ingress-controller:3.5.13
Kong Admin Service
apiVersion: v1
kind: Service
metadata:
name: kong-admin
namespace: <kong-namespace>
spec:
type: ClusterIP
sessionAffinity: None
publishNotReadyAddresses: true
Problem Description
During rolling restarts of Kong proxy pods, newly created Kong dataplane instances intermittently remain in an unconfigured state.
The readiness endpoint returns:
{
"message": "no configuration available (empty configuration present)"
}
The Admin API reports:
{
"configuration_hash": "00000000000000000000000000000000"
}
Key Evidence
EndpointSlices contain all three pods
- 10.16.3.19
- 10.16.3.20
- 10.16.0.26
KIC logs
Fetching last good configuration from gateway clients
{"count":1}
Sending configuration to gateway clients
urls=["https://kong-admin:8444"]
Failed to fetch last good configuration from gateways
HTTP status 404 (message: Not found)
Controller restart recovery
Before restart:
configuration_hash=00000000000000000000000000000000
After restart:
pod becomes ready
configuration_hash becomes non-zero
Questions For Kong Engineering
- Is KIC expected to synchronize configuration independently to every Kong dataplane instance when using a shared Admin Service?
- Can Service-level load balancing contribute to delayed configuration propagation?
- Is this a known limitation in DB-less multi-replica deployments?
- Is Gateway Discovery the recommended topology?
- Why does a dataplane remain at configuration_hash=000… after appearing in EndpointSlices and being reachable via the Admin Service?
This is what I found in controller logs.
2026-09-09T19:41:52Z info setup Initializing Dataplane address Discovery
2026-09-09T19:42:00Z debug Fetching last good configuration from gateway clients {"count":1}
2026-09-09T19:42:00Z error Failed to fetch last good configuration from gateways
HTTP status 404 (Not found)
failed to fetch configuration from ``https://kong-admin:8444
2026-09-09T19:42:00Z debug Reconciling EndpointSlice kong-admin
2026-09-09T19:42:00Z debug Reconciling EndpointSlice kong-proxy
2026-09-09T19:42:00Z debug Sending configuration to gateway clients urls=["``https://kong-admin:8444``"]
2026-09-09T19:42:03Z debug Sending configuration to gateway clients urls=["``https://kong-admin:8444``"]
2026-09-09T19:42:06Z debug Sending configuration to gateway clients urls=["``https://kong-admin:8444``"]
2026-09-09T19:42:09Z debug Sending configuration to gateway clients urls=["``https://kong-admin:8444``"]
2026-09-09T19:42:17Z debug Reconciling EndpointSlice kong-admin
2026-09-09T19:42:17Z debug Reconciling EndpointSlice kong-proxy
2026-09-09T19:52:02Z debug Reconciling EndpointSlice kong-admin
2026-09-09T19:52:02Z debug Reconciling EndpointSlice kong-proxy
2026-09-09T19:52:19Z debug Reconciling EndpointSlice kong-admin
2026-09-09T19:52:19Z debug Reconciling EndpointSlice kong-proxy