Old version of kong ingress is missing permissionos

Hi,

I’am trying to deploy an old version of kong ingress controller (0.8.0) via helm chart but I run into the problem that the ingress controller is missing permissions.

Ingress container logs

W0413 06:31:21.521615       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0413 06:31:21.521943       1 main.go:442] Creating API client for https://10.245.0.1:443
I0413 06:31:21.534489       1 main.go:486] Running in Kubernetes Cluster version v1.20 (v1.20.15) - git (clean) commit 8f1e5bf0b9729a899b8df86249b56e2c74aebc55 - platform linux/amd64
I0413 06:31:21.554906       1 main.go:189] kong version: 2.6.0
I0413 06:31:21.555010       1 main.go:198] Kong datastore: off
E0413 06:31:21.580906       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:22.585289       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:23.590497       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:24.596900       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:25.601854       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:26.605591       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:27.609661       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:28.613140       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:29.618513       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:30.622440       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:31.626627       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:32.633030       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:33.636804       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:34.639811       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:35.644392       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:36.647068       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:37.650975       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:38.656188       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:39.659921       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:40.665819       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:41.670366       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:42.675694       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:43.679496       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:44.685144       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:45.704000       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:46.708121       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:47.711185       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:48.714917       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:49.718255       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope
E0413 06:31:50.722434       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v1.KongCredential: kongcredentials.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-1648710167-kong" cannot list resource "kongcredentials" in API group "configuration.konghq.com" at the cluster scope

I’ve tried to set the permissions on the service account with the following yaml file:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kong-ingress-additional-permissions
rules:
  - apiGroups:
      - configuration.konghq.com
    resources: ["KongCredentials"]
    verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kong-1648710167-kong-additional-permissions
  namespace: kong
subjects:
  - kind: ServiceAccount
    name: kong-1648710167-kong
    namespace: kong
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kong-ingress-additional-permissions

kubectl describe pod

Name:           kong-1648710167-kong-55795545b4-dpbgx
Namespace:      kong
Priority:       0
Node:           pool-test-do-ams3-01-uzoo7/10.133.93.230
Start Time:     Thu, 31 Mar 2022 09:02:50 +0200
Labels:         app=kong-1648710167-kong
                app.kubernetes.io/component=app
                app.kubernetes.io/instance=kong-1648710167
                app.kubernetes.io/managed-by=Helm
                app.kubernetes.io/name=kong
                app.kubernetes.io/version=2.7
                helm.sh/chart=kong-2.7.0
                pod-template-hash=55795545b4
                version=2.7
Annotations:    kuma.io/gateway: enabled
                traffic.sidecar.istio.io/includeInboundPorts: 
Status:         Running
IP:             10.244.2.149
Controlled By:  ReplicaSet/kong-1648710167-kong-55795545b4
Init Containers:
  clear-stale-pid:
    Container ID:  containerd://e3c1b212002b2f87623e97ff649e30b36c49e2a3be5536fc72b46fa944169105
    Image:         kong:2.6
    Image ID:      docker.io/library/kong@sha256:5f819f70e609596a17c858ad6c932492f8aca8e6d0776926e854ba292a46b69a
    Port:          <none>
    Host Port:     <none>
    Command:
      rm
      -vrf
      $KONG_PREFIX/pids
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 31 Mar 2022 09:02:51 +0200
      Finished:     Thu, 31 Mar 2022 09:02:51 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      KONG_ADMIN_ACCESS_LOG:        /dev/stdout
      KONG_ADMIN_ERROR_LOG:         /dev/stderr
      KONG_ADMIN_GUI_ACCESS_LOG:    /dev/stdout
      KONG_ADMIN_GUI_ERROR_LOG:     /dev/stderr
      KONG_ADMIN_LISTEN:            127.0.0.1:8444 http2 ssl
      KONG_CLUSTER_LISTEN:          off
      KONG_DATABASE:                off
      KONG_KIC:                     on
      KONG_LUA_PACKAGE_PATH:        /opt/?.lua;/opt/?/init.lua;;
      KONG_NGINX_WORKER_PROCESSES:  2
      KONG_PLUGINS:                 bundled,ticketengine-auth
      KONG_PORTAL_API_ACCESS_LOG:   /dev/stdout
      KONG_PORTAL_API_ERROR_LOG:    /dev/stderr
      KONG_PORT_MAPS:               80:8000, 443:8443
      KONG_PREFIX:                  /kong_prefix/
      KONG_PROXY_ACCESS_LOG:        /dev/stdout
      KONG_PROXY_ERROR_LOG:         /dev/stderr
      KONG_PROXY_LISTEN:            0.0.0.0:8000, 0.0.0.0:8443 http2 ssl
      KONG_STATUS_LISTEN:           0.0.0.0:8100
      KONG_STREAM_LISTEN:           off
    Mounts:
      /kong_prefix/ from kong-1648710167-kong-prefix-dir (rw)
      /opt/kong/plugins/ticketengine-auth from kong-plugin-ticketengine-auth (ro)
      /tmp from kong-1648710167-kong-tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kong-1648710167-kong-token-7f5dp (ro)
Containers:
  ingress-controller:
    Container ID:   containerd://cc6d8e9e3d0fc1222e82a4a3562c5f0f82d7b1af481b14d2969b0e4cba381e53
    Image:          kong/kubernetes-ingress-controller:0.8.0
    Image ID:       docker.io/kong/kubernetes-ingress-controller@sha256:4bc984a937cefa1aba1cfe283d757e959350abd3143178b44e415dd0d40680ab
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Wed, 13 Apr 2022 08:31:21 +0200
      Finished:     Wed, 13 Apr 2022 08:31:51 +0200
    Ready:          False
    Restart Count:  6154
    Liveness:       http-get http://:10254/healthz delay=5s timeout=5s period=10s #success=1 #failure=3
    Readiness:      http-get http://:10254/healthz delay=5s timeout=5s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:                               kong-1648710167-kong-55795545b4-dpbgx (v1:metadata.name)
      POD_NAMESPACE:                          kong (v1:metadata.namespace)
      CONTROLLER_ELECTION_ID:                 kong-ingress-controller-leader-kong
      CONTROLLER_INGRESS_CLASS:               kong
      CONTROLLER_KONG_ADMIN_TLS_SKIP_VERIFY:  true
      CONTROLLER_KONG_ADMIN_URL:              https://localhost:8444
      CONTROLLER_PUBLISH_SERVICE:             kong/kong-1648710167-kong-proxy
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kong-1648710167-kong-token-7f5dp (ro)
  proxy:
    Container ID:   containerd://633a7f29596ce69bfd69702bfac064768daced4bb4b715084a921e681371c8e8
    Image:          kong:2.6
    Image ID:       docker.io/library/kong@sha256:5f819f70e609596a17c858ad6c932492f8aca8e6d0776926e854ba292a46b69a
    Ports:          8000/TCP, 8443/TCP, 8100/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Running
      Started:      Thu, 31 Mar 2022 09:02:53 +0200
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:status/status delay=5s timeout=5s period=10s #success=1 #failure=3
    Readiness:      http-get http://:status/status delay=5s timeout=5s period=10s #success=1 #failure=3
    Environment:
      KONG_ADMIN_ACCESS_LOG:        /dev/stdout
      KONG_ADMIN_ERROR_LOG:         /dev/stderr
      KONG_ADMIN_GUI_ACCESS_LOG:    /dev/stdout
      KONG_ADMIN_GUI_ERROR_LOG:     /dev/stderr
      KONG_ADMIN_LISTEN:            127.0.0.1:8444 http2 ssl
      KONG_CLUSTER_LISTEN:          off
      KONG_DATABASE:                off
      KONG_KIC:                     on
      KONG_LUA_PACKAGE_PATH:        /opt/?.lua;/opt/?/init.lua;;
      KONG_NGINX_WORKER_PROCESSES:  2
      KONG_PLUGINS:                 bundled,ticketengine-auth
      KONG_PORTAL_API_ACCESS_LOG:   /dev/stdout
      KONG_PORTAL_API_ERROR_LOG:    /dev/stderr
      KONG_PORT_MAPS:               80:8000, 443:8443
      KONG_PREFIX:                  /kong_prefix/
      KONG_PROXY_ACCESS_LOG:        /dev/stdout
      KONG_PROXY_ERROR_LOG:         /dev/stderr
      KONG_PROXY_LISTEN:            0.0.0.0:8000, 0.0.0.0:8443 http2 ssl
      KONG_STATUS_LISTEN:           0.0.0.0:8100
      KONG_STREAM_LISTEN:           off
      KONG_NGINX_DAEMON:            off
    Mounts:
      /kong_prefix/ from kong-1648710167-kong-prefix-dir (rw)
      /opt/kong/plugins/ticketengine-auth from kong-plugin-ticketengine-auth (ro)
      /tmp from kong-1648710167-kong-tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kong-1648710167-kong-token-7f5dp (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kong-1648710167-kong-prefix-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kong-1648710167-kong-tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kong-plugin-ticketengine-auth:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kong-plugin-ticketengine-auth
    Optional:  false
  kong-1648710167-kong-token-7f5dp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kong-1648710167-kong-token-7f5dp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                      From                                 Message
  ----     ------     ----                     ----                                 -------
  Warning  Unhealthy  19m (x13795 over 12d)    kubelet, pool-test-do-ams3-01-uzoo7  Readiness probe failed: Get "http://10.244.2.149:10254/healthz": dial tcp 10.244.2.149:10254: connect: connection refused
  Warning  BackOff    4m41s (x75224 over 12d)  kubelet, pool-test-do-ams3-01-uzoo7  Back-off restarting failed container

Does anyone have a suggestion to get the kong ingress up and running?

Thanks

Does that not clear the issue after a restart? That permission looks correct.

You could alternately try extracting the account and roles from the 0.8 manifest, applying those independent of the chart, and then use that user instead.

Hi traines,

thank you for your reply. The issue does not clear after restart, unfortunately.

Not sure if I understand you correctly. Should I overwrite the deployment.serviceAccount.name property in the kong config helm chart with the user created from the 0.8 manifest?

Thanks

Yep, that’s how you’d tell the chart to use that user instead.