Kong 1.3 - ingress-controller issue

1

Hi everyone,
in our solution in Kubernetes we have applications with an Ingress associated to a KongIngress.
All seemed to work fine, but yesterday for some test I deleted an application (deployment, service, ingress and kongingress) and I recreated it.
The field “Address” in Ingress was empty.
The ingress-kong pod was running, but logs for ingress-controller container stopped some days ago
and Last State was Error with Exit Code 255.
I resolved the issue restarting (delete) the ingress-kong pod, and automatically the Address field in Ingress was recovered.
How could I prevent it? Could I check other logs?

Thanks in advance
Diego

2

It seems like the controller exited but Kubernetes should have restarted it or restarted a new container.

Was there anything interesting the error log?
Can you share the output kubectl describe for the pod?

3

The last log reports a lot of similar info:

I0118 20:55:45.940971 1 controller.go:135] successfully synced configuration to Kong
I0118 20:55:49.821760 1 controller.go:135] successfully synced configuration to Kong

while this is the output of kubectl describe after my kubectl delete pod:

Name:           ingress-kong-669579c865-c95zr
Namespace:      kong
Priority:       0
Node:           ****************
Start Time:     Tue, 21 Jan 2020 17:17:47 +0100
Labels:         app=ingress-kong
              pod-template-hash=669579c865
Annotations:    prometheus.io/port: 9542
                prometheus.io/scrape: true
                traffic.sidecar.istio.io/includeInboundPorts:
Status:         Running
IP:             10.244.0.134
IPs:            <none>
Controlled By:  ReplicaSet/ingress-kong-669579c865
Init Containers:
  wait-for-migrations:
    Container ID:  ****************
    Image:         kong:1.3
    Image ID:      ****************
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      while true; do kong migrations list; if [[ 0 -eq $? ]]; then exit 0; fi; sleep 2;  done;
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 21 Jan 2020 17:17:53 +0100
      Finished:     Tue, 21 Jan 2020 17:17:54 +0100
    Ready:          True
    Restart Count:  0
    Environment:
      KONG_PG_HOST:      ****************
      KONG_PG_PASSWORD:  ****************
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kong-serviceaccount-token-k5dwn (ro)
Containers:
  proxy:
    Container ID:   ****************
    Image:          kong:1.3
    Image ID:       ****************
    Ports:          8000/TCP, 8443/TCP, 9542/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Running
      Started:      Tue, 21 Jan 2020 17:17:56 +0100
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:9001/health delay=30s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:9001/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      KONG_DATABASE:                ****************
      KONG_PG_HOST:                 ****************
      KONG_PG_PASSWORD:             ****************
      KONG_NGINX_WORKER_PROCESSES:  1
      KONG_NGINX_HTTP_INCLUDE:      /kong/servers.conf
      KONG_ADMIN_ACCESS_LOG:        /dev/stdout
      KONG_ADMIN_ERROR_LOG:         /dev/stderr
      KONG_ADMIN_LISTEN:            0.0.0.0:8444
      KONG_PROXY_LISTEN:            0.0.0.0:8000, 0.0.0.0:8443 ssl http2
    Mounts:
      /kong from kong-server-blocks (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kong-serviceaccount-token-k5dwn (ro)
  ingress-controller:
    Container ID:  ****************
    Image:         kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller:0.6.2
    Image ID:      ****************
    Port:          8080/TCP
    Host Port:     0/TCP
    Args:
      /kong-ingress-controller
      --kong-url=http://localhost:8444
      --admin-tls-skip-verify
      --publish-service=kong/kong-proxy
    State:          Running
      Started:      Tue, 21 Jan 2020 17:17:59 +0100
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Tue, 21 Jan 2020 17:17:57 +0100
      Finished:     Tue, 21 Jan 2020 17:17:58 +0100
    Ready:          True
    Restart Count:  1
    Liveness:       http-get http://:10254/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:10254/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-kong-669579c865-c95zr (v1:metadata.name)
      POD_NAMESPACE:  kong (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kong-serviceaccount-token-k5dwn (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kong-server-blocks:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kong-server-blocks
    Optional:  false
  kong-serviceaccount-token-k5dwn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kong-serviceaccount-token-k5dwn
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>
4

Did k8s didn’t automatically replace the dead pod with a new one?
Did the pod actually die or was it in another state?

5

I’m not sure of the status of each containers in the pod before I deleted it, but for sure the pod was running and there was some problem with it.
Deleting and recreating some APIs (deployment, service, ingress and kongingress) without any change in yaml files, the output kubectl get ingress showed me these APIs without address.

Untitled

At that point I deleted manually the pod ingress-kong and after it restarted, the APIs recovered the address.
I hope I was clear enough.

In the output kubectl describe for ingress-kong pod, what does it mean

Last State:     Terminated
  Reason:       Error
  Exit Code:    255
  Started:      Tue, 21 Jan 2020 17:17:57 +0100
  Finished:     Tue, 21 Jan 2020 17:17:58 +0100
Ready:          True
Restart Count:  1

for ingress-controller container?

I’m testing now the same procedure and at the moment I have the same issue.
kubectl logs -f ingress-kong shows me this for my new API :

W0124 14:40:10.079981 1 parser.go:772] service ********* does not have any active endpoints

Many thanks
Diego