Kuma Sidecar missing serviceaccount secret mount - crashloopbackoff occurs

Hi,

I’m playing with using Kuma service mesh with Kong Gateway on a local k3d k8s cluster.

I’ve installed Kuma using its helm chart and I’ve installed a hybrid deployment of Kong Gateway (not ingress controller) using it’s helm chart. I have a control plane in a “kong-system” namespace and a data-plane in an “api” namespace.

I started off with Kuma annotations to disable the sidecar from deploying:

podAnnotations:
  kuma.io/sidecar-injection: disabled

At this stage the data plane started successfully and my example python API was able to be proxied by kong.

When enabling the Kuma sidecar for Kong by adding the following annotations to the Kong Helm values file and uninstalling and reinstalling the Kong data plane deployment

podAnnotations:
  kuma.io/sidecar-injection: enabled
  kuma.io/gateway: enabled

A sidecar Kuma data plane gets created but gets stuck in a CrashLoopBackOff and the logs give the following:

Error: could not read file /var/run/secrets/kubernetes.io/serviceaccount/token: stat /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory

Comparing this to my API deployment where its Kuma sidecar starts perfectly I can see in the kubectl describes that the mount is missing for the service account token for the Kong Data Plane:

   KUMA_CONTROL_PLANE_URL:             https://kuma-control-plane.kuma-system:5678                                                                                                                           │
│       KUMA_DATAPLANE_ADMIN_PORT:          9901                                                                                                                                                                  │
│       KUMA_DATAPLANE_DRAIN_TIME:          30s                                                                                                                                                                   │
│       KUMA_DATAPLANE_MESH:                pensions                                                                                                                                                              │
│       KUMA_DATAPLANE_NAME:                $(POD_NAME).$(POD_NAMESPACE)                                                                                                                                          │
│       KUMA_DATAPLANE_RUNTIME_TOKEN_PATH:  /var/run/secrets/kubernetes.io/serviceaccount/token                                                                                                                   │
│       KUMA_DNS_CORE_DNS_BINARY_PATH:      coredns                                                                                                                                                               │
│       KUMA_DNS_CORE_DNS_EMPTY_PORT:       15054                                                                                                                                                                 │
│       KUMA_DNS_CORE_DNS_PORT:             15053                                                                                                                                                                 │
│       KUMA_DNS_ENABLED:                   true                                                                                                                                                                  │
│       KUMA_DNS_ENVOY_DNS_PORT:            15055                                                                                                                                                                 │
│     Mounts:                               <none>                                                                                                                                                                │
│ Conditions:                                      

Whereas the api sidecar describe has a mount:

  KUMA_DNS_CORE_DNS_BINARY_PATH:      coredns                                                                                                                                                               │
│       KUMA_DNS_CORE_DNS_EMPTY_PORT:       15054                                                                                                                                                                 │
│       KUMA_DNS_CORE_DNS_PORT:             15053                                                                                                                                                                 │
│       KUMA_DNS_ENABLED:                   true                                                                                                                                                                  │
│       KUMA_DNS_ENVOY_DNS_PORT:            15055                                                                                                                                                                 │
│     Mounts:                                                                                                                                                                                                     │
│       /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-p2vq2 (ro)

I’ve tried a google without much luck and now a bit stuck so any advice or hints to debug would be greatly appreciated as to why this mount is missing.

Rob

A bit more digging it looks like

spec:
  automountServiceAccountToken: false

is being set on the Kong Data Plane pod spec which is what seems to be stopping the serviceaccount from mounting for the Kuma sidecar

Is this a necessary setting for the Kong Gateway pod? I’m relatively still new to K8s so probably missing something obvious!

Rob

This is down to the Kong Helm Chart linking the automountServiceAccountToken value to:

 automountServiceAccountToken: {{ .Values.ingressController.enabled }}

Because I was deploying Kong in “full” mode (i.e. data planes and control planes hybrid on K8s) opposed to the ingress controller this was set to false in my values.yaml

I’ll raise an issue on project in GitHub and progress there

Rob

Github Issue: automountServiceAccountToken: false causes crashloop with Kuma DP sidecar · Issue #467 · Kong/charts · GitHub