Error: cannot list resource "kongclusterplugins" in API group "" at the cluster scope

hi, I am seeing these errors after upgrading from KIC 1.3.4 (chart 2.3.0) to 2.0 (chart 2.4.0).
Kong version is 2.0.5 and we did not change it across the upgrade.

E1121 18:31:49.482814       1 reflector.go:138] pkg/mod/ Failed to watch *v1.KongClusterPlugin: failed to list *v1.KongClusterPlugin: is forbidden: User "system:serviceaccount:test:test-service-account" cannot list resource "kongclusterplugins" in API group "" at the cluster scope

I noticed that Kong clusterrole related to kongclusterplugins in chart 2.3.0 was changed to a role in 2.4.0, but I see that the rolebinding is there for the test-service-account, so it shouldn’t be a problem.

I did not update the CRD to chart 2.4.0 version, as this will impact all namespaces, and I am updating just the test namespace. There is a different prod namespace where there is a separate deployment of KIC, that is still on version 1.3.4.

Just wondering if there is something that I’ve overlooked. If someone could help point me in the right direction, I’d really appreciate it. thanks!

Are you using watchNamespaces? Without that, the role should still include the KongClusterPlugin permission, though it should still be a ClusterRole (Roles cannot grant permissions to cluster-scoped resources).

In some version we added additional functionality to create per-namespace roles when using watchNamespaces, and enabling that will remove your ability to use cluster-scoped resources, following user requests to provide an option that didn’t require cluster-level permissions. We’ve since walked back this decision partially, and the latest releases of the chart instead create per-namespace Roles for namespace-scoped resources, but also create a ClusterRole for KongClusterPlugin, IngressClass, etc.

On older versions, you need to delete the KongClusterPlugin CRD (note that this will delete all existing KongClusterPlugin resources–you should instead upgrade to version that lets you disable the controller) to disable the controller’s attempts to fetch it. On 2.x and above you can set --enable-controller-kongclusterplugin=false instead.

You should, in general, not rely on namespaces for prod/non-prod segmentation (or even segmentation between multiple non-prod environments). Cluster-scoped resources and the like make doing so all but impossible. If you don’t have the option of running additional full-size clusters, I’d recommend instead testing in KIND or similar. We provide GitHub - Kong/kubernetes-testing-framework: Golang Integration Testing Framework For Kubernetes APIs and Controllers. as a KIND wrapper, to simplify common tasks (in particular, setting up MetalLB for LoadBalancer support) useful for testing Kong instances.