Error: cannot list resource "kongclusterplugins" in API group "configuration.konghq.com" at the cluster scope

Kubernetes
1

hi, I am seeing these errors after upgrading from KIC 1.3.4 (chart 2.3.0) to 2.0 (chart 2.4.0).
Kong version is 2.0.5 and we did not change it across the upgrade.

E1121 18:31:49.482814       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:167: Failed to watch *v1.KongClusterPlugin: failed to list *v1.KongClusterPlugin: kongclusterplugins.configuration.konghq.com is forbidden: User "system:serviceaccount:test:test-service-account" cannot list resource "kongclusterplugins" in API group "configuration.konghq.com" at the cluster scope

I noticed that Kong clusterrole related to kongclusterplugins in chart 2.3.0 was changed to a role in 2.4.0, but I see that the rolebinding is there for the test-service-account, so it shouldn’t be a problem.

I did not update the CRD to chart 2.4.0 version, as this will impact all namespaces, and I am updating just the test namespace. There is a different prod namespace where there is a separate deployment of KIC, that is still on version 1.3.4.

Just wondering if there is something that I’ve overlooked. If someone could help point me in the right direction, I’d really appreciate it. thanks!

2

Are you using watchNamespaces? Without that, the role should still include the KongClusterPlugin permission, though it should still be a ClusterRole (Roles cannot grant permissions to cluster-scoped resources).

In some version we added additional functionality to create per-namespace roles when using watchNamespaces, and enabling that will remove your ability to use cluster-scoped resources, following user requests to provide an option that didn’t require cluster-level permissions. We’ve since walked back this decision partially, and the latest releases of the chart instead create per-namespace Roles for namespace-scoped resources, but also create a ClusterRole for KongClusterPlugin, IngressClass, etc.

On older versions, you need to delete the KongClusterPlugin CRD (note that this will delete all existing KongClusterPlugin resources–you should instead upgrade to version that lets you disable the controller) to disable the controller’s attempts to fetch it. On 2.x and above you can set --enable-controller-kongclusterplugin=false instead.

You should, in general, not rely on namespaces for prod/non-prod segmentation (or even segmentation between multiple non-prod environments). Cluster-scoped resources and the like make doing so all but impossible. If you don’t have the option of running additional full-size clusters, I’d recommend instead testing in KIND or similar. We provide GitHub - Kong/kubernetes-testing-framework: Golang Integration Testing Framework For Kubernetes APIs and Controllers. as a KIND wrapper, to simplify common tasks (in particular, setting up MetalLB for LoadBalancer support) useful for testing Kong instances.

3

Thanks for getting back to me about this. Yes I am using watchNamespaces when upgrading to KIC 2.0

For KIC 1.3.4, this is what I have in the values.yaml to watch just the namespace where the KIC is running:

ingressController:
  enabled: true
  image:
    repository: kong/kubernetes-ingress-controller
    tag: "1.3"
    pullPolicy: Always
  args: [--watch-namespace=$(POD_NAMESPACE)]

Upgrading from KIC 1.3.4 to KIC 2.0.5, I am updating the values.yaml to be:

ingressController:
  enabled: true
  image:
    repository: kong/kubernetes-ingress-controller
    tag: "2.0.5"
    pullPolicy: Always
  watchNamespaces: [test]

The reason that I am making this change from args: [--watch-namespace=$(POD_NAMESPACE)] to watchNamespaces: [test], during the upgrade is that I am inferring from the documentation that args: [--watch-namespace=$(POD_NAMESPACE)] shouldn’t be used any more, and instead use watchNamespaces: [test]

Please let me know if this is not the case.

If args: [--watch-namespace=$(POD_NAMESPACE)] is supported in KIC 2.0.5, I’ll continue to use it.

We do use KongClusterPlugin, and we want to move away from that in KIC 2.0.5, after the upgrade of KIC 1.3.4 to KIC 2.0.5 is successful. The upgrade is a little tricky because we have a prod namespace, and we are making our best attempt to upgrade it while making as little change as possible to prevent any production outages.

Yes we are relying on namespaces for prod/staging/dev/test segmentation. We have KIC deployment in each of the namespaces.

4

You’ll want to set --enable-controller-kongclusterplugin=false or get to chart version 2.12. Prior to 2.12 the chart does not create a ClusterRole when using watchNamespaces.

5

Hi traines,

Just a couple of brief questions:
Is it possible to upgrade directly from chart 2.3.0 with KIC 1.3.4 to chart 2.12.0 with KIC 2.0?
Is args: [--watch-namespace=$(POD_NAMESPACE)] is supported in KIC 2.0.5?

6

Hard to say for sure, but charts/UPGRADE.md at main · Kong/charts · GitHub doesn’t list anything that’d obviously require you to pause at certain versions. You cannot go directly to KIC versions after 2.0.x, since you need to apply the 2.0.x CRDs before applying CRDs for later versions.

--watch-namespace is supported on both versions. 2.0 expands its functionality to accept multiple values.