Error: Trying to recreate duplicated resources after upgraded KIC from 1.3 to 2.0

Kubernetes
1

I did the suggested procedure to migrate from:
https://github.com/Kong/charts/blob/main/charts/kong/UPGRADE.md#240

the current environment:
Kubernetes: EKS 1.21
Database: PostgresSQL engine: 12.5 managed by AWS
Kong Chart: from 2.3.0 to 2.4.0

Issue:
After upgrading KIC 1.3 to 2.0 following suggested procedure where both KIC were not running simultaneously during upgrade due to known issue with postgres, everything went smoothly but after a restart of the pods, I started to see in the ingress-controller logs the following error:

time="2022-12-02T12:11:51Z" level=error msg="could not update kong admin" error="10 errors occurred:\n\twhile processing event: {Create} consumer

this error happens with consumers, services, paths and plugins.
I removed by admin api the duplicated resources and the ingress-controller regenerated them, one thing I observed the duplicated resources was created with NULL tags after restart and after I removed them, the resources were recreated with the correct tag “managed-by-ingress-controller”
Every time the pods restart, this error happens

Any suggestion or configuration that can check?

Thanks

2

If you proceed further through the upgrades, to at least 2.2.1, do you still see this issue? I don’t know of anything quite like what’s described, but there was a significant issue in the initial 2.x releases where the controller handled leader election improperly and would make incorrect changes, and it’s possible that’s related.

If upgrading doesn’t help:

To confirm, does NULL mean the tag set is empty or there’s a single NULL tag?

Do you see any admin API access logs indicating a change from a user-agent other than Go-http-client/1.1?

Are you able to pinpoint exactly when the tags are being modified and share controller logs? Debug-level controller logs should include diff information showing when it’s adding or removing tags (or other settings).

3

thanks @traines for the suggestions, I’m going to upgrade to 2.2.1 and check if the issue is gone.
Right now I can answer one of the questions:
NULL means empty, the following example is one of the consumers created without tags
image
And also I checked the postgres table to confirm
image

I’ll collect the logs and share it.

Thanks