Error: Trying to recreate duplicated resources after upgraded KIC from 1.3 to 2.0

I did the suggested procedure to migrate from:

the current environment:
Kubernetes: EKS 1.21
Database: PostgresSQL engine: 12.5 managed by AWS
Kong Chart: from 2.3.0 to 2.4.0

After upgrading KIC 1.3 to 2.0 following suggested procedure where both KIC were not running simultaneously during upgrade due to known issue with postgres, everything went smoothly but after a restart of the pods, I started to see in the ingress-controller logs the following error:

time="2022-12-02T12:11:51Z" level=error msg="could not update kong admin" error="10 errors occurred:\n\twhile processing event: {Create} consumer

this error happens with consumers, services, paths and plugins.
I removed by admin api the duplicated resources and the ingress-controller regenerated them, one thing I observed the duplicated resources was created with NULL tags after restart and after I removed them, the resources were recreated with the correct tag “managed-by-ingress-controller
Every time the pods restart, this error happens

Any suggestion or configuration that can check?


If you proceed further through the upgrades, to at least 2.2.1, do you still see this issue? I don’t know of anything quite like what’s described, but there was a significant issue in the initial 2.x releases where the controller handled leader election improperly and would make incorrect changes, and it’s possible that’s related.

If upgrading doesn’t help:

To confirm, does NULL mean the tag set is empty or there’s a single NULL tag?

Do you see any admin API access logs indicating a change from a user-agent other than Go-http-client/1.1?

Are you able to pinpoint exactly when the tags are being modified and share controller logs? Debug-level controller logs should include diff information showing when it’s adding or removing tags (or other settings).

thanks @traines for the suggestions, I’m going to upgrade to 2.2.1 and check if the issue is gone.
Right now I can answer one of the questions:
NULL means empty, the following example is one of the consumers created without tags
And also I checked the postgres table to confirm

I’ll collect the logs and share it.