the current environment:
Kubernetes: EKS 1.21
Database: PostgresSQL engine: 12.5 managed by AWS
Kong Chart: from 2.3.0 to 2.4.0
Issue:
After upgrading KIC 1.3 to 2.0 following suggested procedure where both KIC were not running simultaneously during upgrade due to known issue with postgres, everything went smoothly but after a restart of the pods, I started to see in the ingress-controller logs the following error:
time="2022-12-02T12:11:51Z" level=error msg="could not update kong admin" error="10 errors occurred:\n\twhile processing event: {Create} consumer
this error happens with consumers, services, paths and plugins.
I removed by admin api the duplicated resources and the ingress-controller regenerated them, one thing I observed the duplicated resources was created with NULL tags after restart and after I removed them, the resources were recreated with the correct tag “managed-by-ingress-controller”
Every time the pods restart, this error happens
If you proceed further through the upgrades, to at least 2.2.1, do you still see this issue? I don’t know of anything quite like what’s described, but there was a significant issue in the initial 2.x releases where the controller handled leader election improperly and would make incorrect changes, and it’s possible that’s related.
If upgrading doesn’t help:
To confirm, does NULL mean the tag set is empty or there’s a single NULL tag?
Do you see any admin API access logs indicating a change from a user-agent other than Go-http-client/1.1?
Are you able to pinpoint exactly when the tags are being modified and share controller logs? Debug-level controller logs should include diff information showing when it’s adding or removing tags (or other settings).
thanks @traines for the suggestions, I’m going to upgrade to 2.2.1 and check if the issue is gone.
Right now I can answer one of the questions:
NULL means empty, the following example is one of the consumers created without tags
And also I checked the postgres table to confirm
