Kong-controlplane is always waiting to start

Kubernetes
1

Hi all,

Need help here. Followed the [Installation Options - v3.5.x | Kong Docs](Kong Kubernetes Installation) to deploy Kong API gateway backed up with Postgres to Kubernetes. After executing the step of kubectl -n kong apply -f kong-control-plane-postgres.yaml, then try to
kubectl get pods -n kong, it returns the following pods which is always in Initializing status

NAME READY STATUS RESTARTS AGE
kong-control-plane-6c8b4c89b7-4cw6g 0/1 Init:0/2 0 19s
kong-control-plane-bootstrap-bgfmg 0/1 Init:0/1 0 19s
postgres-bztrl 1/1 Running 0 84s

Try to debug the container through kubectl logs kong-control-plane-bootstrap-bgfmg wait-for-postgres -n kong, it shows the following error:

nc: bad address ‘postgres.kong.svc’
waiting for db

Any problem with the deployment? Please help

3

Try your way to change the command from “nc -zv” to “nslookup”, the db still not reachable throwing the error in logs. Any idea how to troubleshoot?

waiting for db
;; connection timed out; no servers could be reached

4

What do the postgres logs say?

That repository includes automated nightly tests so I feel fairly confident the Kong docker image / k8s manifests are correct as they are passing ( https://travis-ci.com/Kong/kong-dist-kubernetes )

5

Hi @hutchic,

Followed the documentation steps to run the following:

kubectl apply -f kong-namespace.yaml
./setup_certificate.sh
kubectl create -f postgres.yaml
kubectl -n kong apply -f kong-control-plane-postgres.yaml

Execute kubectl describe pods -n kong:

Name: kong-control-plane-6c8b4c89b7-wjmnj
Namespace: kong
Priority: 0
Node: awl-mv-ibo-dev/10.20.20.214
Start Time: Thu, 17 Oct 2019 19:57:25 +0800
Labels: app=kong-control-plane
pod-template-hash=6c8b4c89b7
Annotations: cni.projectcalico.org/podIP: 192.168.116.232/32
k8s.konghq.com/sidecar-inject: false
prometheus.io/port: 8001
prometheus.io/scrape: true
Status: Pending
IP: 192.168.116.232
Controlled By: ReplicaSet/kong-control-plane-6c8b4c89b7
Init Containers:
wait-for-postgres:
Container ID: docker://984034bf57798a2bf7969c06c638d81a340a06861d60e6622566dc6a574f5431
Image: busybox:latest
Image ID: docker-pullable://docker.io/busybox@sha256:954e1f01e80ce09d0887ff6ea10b13a812cb01932a0781d6b0cc23f743a874fd
Port:
Host Port:
Command:
/bin/sh
-c
until nc -zv $KONG_PG_HOST $KONG_PG_PORT -w1; do echo ‘waiting for db’; sleep 1; done
State: Running
Started: Thu, 17 Oct 2019 19:57:27 +0800
Ready: False
Restart Count: 0
Environment:
KONG_PG_PORT: 5432
KONG_PG_HOST: postgres.kong.svc
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kong-token-rjwsw (ro)
kong-migration-up:
Container ID:
Image: kong
Image ID:
Port:
Host Port:
Command:
/bin/sh
-c
kong migrations up && kong migrations finish
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
KONG_PG_PASSWORD: kong
KONG_PG_HOST: postgres.kong.svc
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kong-token-rjwsw (ro)
Containers:
kong-control-plane:
Container ID:
Image: kong
Image ID:
Ports: 8001/TCP, 8444/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Liveness: http-get http://:8001/status delay=30s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:8001/status delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
KONG_PG_PASSWORD: kong
KONG_PG_HOST: postgres.kong.svc
KONG_LOG_LEVEL: notice
KONG_ADMIN_ACCESS_LOG: /dev/stdout
KONG_PROXY_ERROR_LOG: /dev/stderr
KONG_ADMIN_ERROR_LOG: /dev/stderr
KONG_ADMIN_LISTEN: 0.0.0.0:8001,0.0.0.0:8444 ssl
KONG_PROXY_LISTEN: off
KONG_ADMIN_SSL_CERT: /api-server-cert/tls.crt
KONG_ADMIN_SSL_CERT_KEY: /api-server-cert/tls.key
Mounts:
/api-server-cert from api-server-cert (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kong-token-rjwsw (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
api-server-cert:
Type: Secret (a volume populated by a Secret)
SecretName: kong-control-plane.kong.svc
Optional: false
kong-token-rjwsw:
Type: Secret (a volume populated by a Secret)
SecretName: kong-token-rjwsw
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message

Normal Scheduled 5m20s default-scheduler Successfully assigned kong/kong-control-plane-6c8b4c89b7-wjmnj to awl-mv-ibo-dev
Normal Pulled 5m19s kubelet, awl-mv-ibo-dev Container image “busybox:latest” already present on machine
Normal Created 5m19s kubelet, awl-mv-ibo-dev Created container wait-for-postgres
Normal Started 5m18s kubelet, awl-mv-ibo-dev Started container wait-for-postgres

Name: kong-control-plane-bootstrap-2n5l2
Namespace: kong
Priority: 0
Node: awl-mv-ibo-dev/10.20.20.214
Start Time: Thu, 17 Oct 2019 19:57:25 +0800
Labels: app=kong-control-plane
controller-uid=73ef02f6-c76a-4923-b3a3-3e7c0c295a43
job-name=kong-control-plane-bootstrap
Annotations: cni.projectcalico.org/podIP: 192.168.116.233/32
Status: Pending
IP: 192.168.116.233
Controlled By: Job/kong-control-plane-bootstrap
Init Containers:
wait-for-postgres:
Container ID: docker://eec396803eace6268058aabd1c79a049dcf8038dc779556c75e1f4a93b7b9353
Image: busybox:latest
Image ID: docker-pullable://docker.io/busybox@sha256:954e1f01e80ce09d0887ff6ea10b13a812cb01932a0781d6b0cc23f743a874fd
Port:
Host Port:
Command:
/bin/sh
-c
until nc -zv $KONG_PG_HOST $KONG_PG_PORT -w1; do echo ‘waiting for db’; sleep 1; done
State: Running
Started: Thu, 17 Oct 2019 19:57:27 +0800
Ready: False
Restart Count: 0
Environment:
KONG_PG_PORT: 5432
KONG_PG_HOST: postgres.kong.svc
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-f2c7v (ro)
Containers:
kong-migration-boostrap:
Container ID:
Image: kong
Image ID:
Port:
Host Port:
Command:
kong
migrations
bootstrap
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
KONG_PG_PASSWORD: kong
KONG_PG_HOST: postgres.kong.svc
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-f2c7v (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-f2c7v:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-f2c7v
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message

Normal Scheduled 5m20s default-scheduler Successfully assigned kong/kong-control-plane-bootstrap-2n5l2 to awl-mv-ibo-dev
Normal Pulled 5m18s kubelet, awl-mv-ibo-dev Container image “busybox:latest” already present on machine
Normal Created 5m18s kubelet, awl-mv-ibo-dev Created container wait-for-postgres
Normal Started 5m18s kubelet, awl-mv-ibo-dev Started container wait-for-postgres

Name: postgres-wzqm7
Namespace: kong
Priority: 0
Node: awl-mv-ibo-dev/10.20.20.214
Start Time: Thu, 17 Oct 2019 19:57:13 +0800
Labels: app=postgres
Annotations: cni.projectcalico.org/podIP: 192.168.116.231/32
Status: Running
IP: 192.168.116.231
Controlled By: ReplicationController/postgres
Containers:
postgres:
Container ID: docker://fa72c4e78480d81e61167a0c7a8db10d2f4f6a347d78d866fe3e333525d63564
Image: postgres:9.6
Image ID: docker-pullable://docker.io/postgres@sha256:e874e4ecbc8f9846027a4132f32a41700e70ac6e0f804ad26fec5b3ab2cd6cff
Port: 5432/TCP
Host Port: 0/TCP
State: Running
Started: Thu, 17 Oct 2019 19:57:15 +0800
Ready: True
Restart Count: 0
Environment:
POSTGRES_USER: kong
POSTGRES_PASSWORD: kong
POSTGRES_DB: kong
PGDATA: /var/lib/postgresql/data/pgdata
Mounts:
/var/lib/postgresql/data from pg-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-f2c7v (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
pg-data:
Type: EmptyDir (a temporary directory that shares a pod’s lifetime)
Medium:
SizeLimit:
default-token-f2c7v:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-f2c7v
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message

Normal Scheduled 5m32s default-scheduler Successfully assigned kong/postgres-wzqm7 to awl-mv-ibo-dev
Normal Pulled 5m31s kubelet, awl-mv-ibo-dev Container image “postgres:9.6” already present on machine
Normal Created 5m31s kubelet, awl-mv-ibo-dev Created container postgres
Normal Started 5m30s kubelet, awl-mv-ibo-dev Started container postgres

Running the kubectl logs -f postgres-wzqm7 -n kong

/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

LOG: received fast shutdown request
LOG: aborting any active transactions
LOG: autovacuum launcher shutting down
LOG: shutting down
waiting for server to shut down…LOG: database system is shut down
done
server stopped

PostgreSQL init process complete; ready for start up.

LOG: database system was shut down at 2019-10-17 11:57:18 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started

I can check the postgres is starting successfully. The waiting-for-postgres init containers seems not able to connect to postgres. Please advice the necessary steps to troubleshoot.

Thanks

6

The kong control plane manifests run a job to bootstrap the datastore can you check if it ran / was successful

kubectl get job -n kong

My full debug:

kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.8", GitCommit:"a89f8c11a5f4f132503edbc4918c98518fd504e3", GitTreeState:"clean", BuildDate:"2019-04-23T04:52:31Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-25T23:41:27Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

git rev-parse --short HEAD
40d9ef2

kubectl apply -f kong-namespace.yaml
./setup_certificate.sh
kubectl create -f postgres.yaml
kubectl get all -n kong #waited for postgres to be ready
kubectl -n kong apply -f kong-control-plane-postgres.yaml
kubectl get all -n kong

If you’re still having difficulty swap the init container with something you’re comfortable debugging from (busybox, ubuntu etc) change the command to tail -f /dev/null and then kubectl exec into it and try and determine why it can’t reach postgrest