`KongPlugin` w/ `KongClusterPlugin` (oauth2) strange behaviour when combined

Kubernetes
1

I’m facing some strange behaviour with Kong Ingress Controller when trying to create a very important use case for me. It seems that when applying a namespaced oauth plugin in an Ingress, it automatically makes KongClusterPlugins global and universal, even without the global flag.

What I want is to have different credentials with oauth2 (client ID and Secrets) that can combine different scopes so that we can differentiate access levels depending on the REST Clients

Example (considering that each API has one scope):

  • Consumer A will be able to use APIs with scopes A,B,C
  • Consumer B will be able to use APIs with scopes B,C,D

This is what I have done:

KongConsumers are omitted for the sake of brevity.

Plugin 1 configuration:

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: "plugin2"
  namespace: ${kubernetes_secret.secret2.metadata.0.namespace}
  annotations:
    kubernetes.io/ingress.class: "kong"
config:
  accept_http_if_already_terminated: true
  enable_authorization_code: false
  enable_client_credentials: true
  mandatory_scope: true
  provision_key: <autogenerated>
  scopes:
    - echo2
  token_expiration: 7200
consumerRef: consumer2
plugin: oauth2
protocols:
  - http
  - https

Plugin 2 Configuration

apiVersion: configuration.konghq.com/v1  
kind: KongPlugin  
metadata:  
  name: "plugin2"  
  namespace: ${kubernetes_secret.secret2.metadata.0.namespace}  
  annotations:  
    kubernetes.io/ingress.class: "kong"  
config:  
  accept_http_if_already_terminated: true  
  enable_authorization_code: false  
  enable_client_credentials: true  
  mandatory_scope: true  
  provision_key: <autogenerated>  
  scopes:  
    - echo2  
  token_expiration: 7200  
consumerRef: consumer2  
plugin: oauth2  
protocols:  
  - http  
  - https

Ingress 1 and Ingress2 (namespaced) have the following annotations:

"konghq.com/plugins" = "plugin1"
and 
"konghq.com/plugins" = "plugin2"

Everything works normally, and each ingress has its own scope, and each login works for 1 scope alone.

To express the use case described above, I want to create a cluster plugin:

apiVersion: configuration.konghq.com/v1  
kind: KongClusterPlugin  
metadata:  
  annotations:  
    kubernetes.io/ingress.class: kong  
  name: cluster-plugin-oauth2  
config:  
  accept_http_if_already_terminated: true  
  enable_authorization_code: false  
  enable_client_credentials: true  
  mandatory_scope: true  
  provision_key: <autogenerated>  
  scopes:  
    - echo1  
    - echo2  
  token_expiration: 7200  
consumerRef: global-api-consumer  
plugin: oauth2  
protocols:  
  - http  
  - https

Note the absence of the global annotation

And I change the annotation in one of the ingresses:

"konghq.com/plugins" = "plugin1,cluster-plugin-oauth2"

now, Kong throws an error while reconciling.

kong kong-kong-578d44c5f7-cqzk9 ingress-controller time="2023-09-22T16:17:28Z" level=error msg="could not update kong admin" error="performing update for https://localhost:8444 failed: inserting plugins into state: inserting plugin oauth2 for route echonamespace2.echo2.echoserver2-service..80: entity already exists" subsystem=dataplane-synchronizer

If I remove the namespaced plugins from the annotation and use only the cluster-plugin-oauth2, then it works as expected, without warnings about duplication

Based on this, I have two questions:

  • Is the above normal? The documentation states here: Using Plugin Resources - v2.11.x | Kong Docs that “Although not shown in this guide, you can also apply a KongClusterPlugin to resources using konghq.com/pluginsannotations, to reuse plugin configurations across namespaces”
    • It seems that when applying a namespaced oauth plugin in an Ingress, it automatically makes KongClusterPlugins global and universal, even without the global flag.
  • Is there any other way for me to achieve what I need?