Kong Ingress Controller >= 0.7 error "consumer":"value must be null propagate to all cluster

Hi,

My installation is consist of Kong 1.3.x + Ingress Controller 0.6.x with custom plugin for decrypting access token (JWT) from my organization OAuth server (called GO-ID) inside GKE .

Later on, i upgrade to Kong 1.5.x + Ingress Controller 0.7.x for grpc support + Istio service mesh. I got this error on ingress controller container and looks like it scanning the whole cluster

    [ingress-kong-5b8cbf9959-kfrxc ingress-controller] W0622 00:13:23.630702       1 parser.go:339] Deprecated KongCredential in use, please use secret-based credentials. KongCredential resource will be removed in future.                                                         
    [ingress-kong-5b8cbf9959-kfrxc ingress-controller] E0622 00:13:23.779074       1 controller.go:119] unexpected failure updating Kong configuration:                                                                                                                               
    [ingress-kong-5b8cbf9959-kfrxc ingress-controller] posting new config to /config: 400 Bad Request {"fields":{"plugins":[{"consumer":"value must be null"},{"consumer":"value must be null"},{"consumer":"value must be null"},{"consumer":"value must be null"}]},"name":"invalid declarative configuration","code":14,"message":"declarative config is invalid: {plugins={{consumer=\"value must be null\"},{consumer=\"value must be null\"},{consumer=\"value must be null\"},{consumer=\"value must be null\"}}}"}                                              
    [ingress-kong-5b8cbf9959-kfrxc ingress-controller] W0622 00:13:23.779102       1 queue.go:112] requeuing kong/sh.helm.release.v1.redis-kong.v5, err posting new config to /config: 400 Bad Request {"fields":{"plugins":[{"consumer":"value must be null"},{"consumer":"value must be null"},{"consumer":"value must be null"},{"consumer":"value must be null"}]},"name":"invalid declarative configuration","code":14,"message":"declarative config is invalid: {plugins={{consumer=\"value must be null\"},{consumer=\"value must be null\"},{consumer=\"value must be null\"},{consumer=\"value must be null\"}}}"}                                                                                      
    [ingress-kong-5b8cbf9959-kfrxc ingress-controller] W0622 00:13:26.964001       1 parser.go:339] Deprecated KongCredential in use, please use secret-based credentials. KongCredential resource will be removed in future.                                                         
    [ingress-kong-5b8cbf9959-kfrxc ingress-controller] E0622 00:13:27.121270       1 controller.go:119] unexpected failure updating Kong configuration:                                                                                                                               
    [ingress-kong-5b8cbf9959-kfrxc ingress-controller] posting new config to /config: 400 Bad Request {"fields":{"plugins":[{"consumer":"value must be null"},{"consumer":"value must be null"},{"consumer":"value must be null"},{"consumer":"value must be null"}]},"name":"invalid declarative configuration","code":14,"message":"declarative config is invalid: {plugins={{consumer=\"value must be null\"},{consumer=\"value must be null\"},{consumer=\"value must be null\"},{consumer=\"value must be null\"}}}"} 
    [ingress-kong-5b8cbf9959-kfrxc ingress-controller] W0622 00:13:27.121302       1 queue.go:112] requeuing 3pc-web-qa/bookinfo-renew-service-dev, err posting new config to /config: 400 Bad Request {"fields":{"plugins":[{"consumer":"value must be null"},{"consumer":"value must be null"},{"consumer":"value must be null"},{"consumer":"value must be null"}]},"name":"invalid declarative configuration","code":14,"message":"declarative config is invalid: {plugins={{consumer=\"value must be null\"},{consumer=\"value must be null\"},{consumer=\"value must be null\"},{consumer=\"value must be null\"}}}"} 
    [ingress-kong-5b8cbf9959-kfrxc ingress-controller] W0622 00:13:30.297474       1 parser.go:339] Deprecated KongCredential in use, please use secret-based credentials. KongCredential resource will be removed in future. 
    [ingress-kong-5b8cbf9959-kfrxc ingress-controller] E0622 00:13:30.463852       1 controller.go:119] unexpected failure updating Kong configuration:  
    [ingress-kong-5b8cbf9959-kfrxc ingress-controller] posting new config to /config: 400 Bad Request {"fields":{"plugins":[{"consumer":"value must be null"},{"consumer":"value must be null"},{"consumer":"value must be null"},{"consumer":"value must be null"}]},"name":"invalid declarative configuration","code":14,"message":"declarative config is invalid: {plugins={{consumer=\"value must be null\"},{consumer=\"value must be null\"},{consumer=\"value must be null\"},{consumer=\"value must be null\"}}}"} 
    [ingress-kong-5b8cbf9959-kfrxc ingress-controller] W0622 00:13:30.464068       1 queue.go:112] requeuing 3pc-web-qa/default-token-wfzj6, err posting new config to /config: 400 Bad Request {"fields":{"plugins":[{"consumer":"value must be null"},{"consumer":"value must be null"},{"consumer":"value must be null"},{"consumer":"value must be null"}]},"name":"invalid declarative configuration","code":14,"message":"declarative config is invalid: {plugins={{consumer=\"value must be null\"},{consumer=\"value must be null\"},{consumer=\"value must be null\"},{consumer=\"value must be null\"}}}"} 

    [ingress-kong-5b8cbf9959-kfrxc ingress-controller] W0622 00:06:56.964023       1 parser.go:339] Deprecated KongCredential in use, please use secret-based credentials. KongCredential resource will be removed in future.                                                         
    [ingress-kong-5b8cbf9959-kfrxc ingress-controller] E0622 00:06:57.106211       1 controller.go:119] unexpected failure updating Kong configuration:                                                                                                                               
    [ingress-kong-5b8cbf9959-kfrxc ingress-controller] posting new config to /config: 400 Bad Request {"fields":{"plugins":[{"consumer":"value must be null"},{"consumer":"value must be null"},{"consumer":"value must be null"},{"consumer":"value must be null"}]},"name":"invalid declarative configuration","code":14,"message":"declarative config is invalid: {plugins={{consumer=\"value must be null\"},{consumer=\"value must be null\"},{consumer=\"value must be null\"},{consumer=\"value must be null\"}}}"}                                              
    [ingress-kong-5b8cbf9959-kfrxc ingress-controller] W0622 00:06:57.106293       1 queue.go:112] requeuing kube-system/istio.pv-protection-controller, err posting new config to /config: 400 Bad Request {"fields":{"plugins":[{"consumer":"value must be null"},{"consumer":"value must be null"},{"consumer":"value must be null"},{"consumer":"value must be null"}]},"name":"invalid declarative configuration","code":14,"message":"declarative config is invalid: {plugins={{consumer=\"value must be null\"},{consumer=\"value must be null\"},{consumer=\"value must be null\"},{consumer=\"value must be null\"}}}"}                                                                                 
    [ingress-kong-5b8cbf9959-kfrxc ingress-controller] W0622 00:07:00.297441       1 parser.go:339] Deprecated KongCredential in use, please use secret-based credentials. KongCredential resource will be removed in future.                                                         
    [ingress-kong-5b8cbf9959-kfrxc ingress-controller] E0622 00:07:00.449651       1 controller.go:119] unexpected failure updating Kong

This is the manifest related to the custom plugin

---
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: key-auth-plugin
  namespace: kong
plugin: key-auth
---
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: goid-consumer
  namespace: kong
  annotations:
    plugins.konghq.com: key-auth-plugin
username: goid-consumer
---
apiVersion: configuration.konghq.com/v1
kind: KongCredential
metadata:
  name: goid-apikey
  namespace: kong
consumerRef: goid-consumer
type: key-auth
config:
  key: <redacted>
---
apiVersion: v1
kind: Service
metadata:
  name: kong-admin
  namespace: kong
spec:
  type: NodePort
  ports:
    - name: admin
      port: 8001
      protocol: TCP
      targetPort: 8001
  selector:
    app: ingress-kong
---
apiVersion: configuration.konghq.com/v1
kind: KongIngress
metadata:
  name: kong-admin-goid-ingress-route
  namespace: kong
proxy:
  path: <redacted>
route:
  strip_path: false
  methods:
    - PUT
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kong-admin-goid-ingress
  namespace: kong
  annotations:
    configuration.konghq.com: kong-admin-goid-ingress-route
    plugins.konghq.com: key-auth-plugin
    kubernetes.io/ingress.class: "kong"
spec:
  rules:
    - host:<redacted>
      http:
        paths:
          - path: <redacted>
            backend:
              serviceName: kong-admin
              servicePort: 8001

some of the consumer

---
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: key-auth-plugin
  namespace: <redacted>
plugin: key-auth
---
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: goid-consumer
  namespace: <redacted>
  annotations:
    plugins.konghq.com: key-auth-plugin
username: goid-consumer
---
apiVersion: configuration.konghq.com/v1
kind: KongCredential
metadata:
  name: goid-apikey
  namespace: <redacted>
consumerRef: goid-consumer
type: key-auth
config:
  key: << redacted >>

I’m quite confused why the ingress controller scan why whole cluster for consumer.

Please remove the plugins.konghq.com annotation from KongConsumer resource.