Ingress-controller error after upgrade to kong-2.8.3 "UNIQUE violation detected on.."

Kubernetes
1

Hi, we upgraded Kong from v2.0.5 to v2.8.3 and following errors started occurring on the ingress controller:

level=error msg="could not update kong admin" 
error="10 errors occurred:\n\twhile processing event: {Create} upstream 
p-api.staging.443.svc failed: HTTP status 409 
(message: \"UNIQUE violation detected on '{name=\\\"p-api.staging.443.svc\\\"}'\")\n\twhile 
processing event: {Create} upstream service-m.staging.443.svc failed: HTTP status 409 (message: 
\"UNIQUE violation detected on '{name=\\\"service-m.staging.443.svc\\\"}'\")

Some additional details:
We are running Kong v2.8.3 with Kong ingress controller v2.1 upgraded with Helm chart kong-2.7.0.
We upgraded from Kong 2.0.5 with Kong Ingress controller v2.0 installed with Helm chart version 2.4.0 on GKE v1.22. We use postgres as the backend database. We weren’t seeing this error before the upgrade.

We’d appreciate any advice in how to resolve this issue. thanks!

2

Hi @aj99 ,

can you please explain when exactly you were getting this error? is it during the deployment of new upstream ?

BTW, instead of waiting for your answer, what I can say is, this type of error comes when we try to create an entity (in this case upstream) and its id (e.g. f3e2c5*****29) is already present in the DB and allocated to an upstream with the same or different name.

The solution is to delete that upstream through admin API or manager and then deploy/create new or same upstream again.

3

Thanks @Sachin_Ghumbre for your response.
We are seeing this error constantly in the logs for all the existing upstream. Should we delete the services and routes, or just the services?

4

As I have less exposure to your kong configuration, I suggest you can try deleting one error specific upstream and see if error disappears for that specific upstream. Make sure it will not create downtime for your environment.

5

Deleting the upstream kubernetes service didn’t work.
I’ve also tried:

  1. deleting the Kong route and service
  2. deleting the Ingress and KongIngress and then recreating the Ingress and KongIngress

I am still seeing constant (once every second or two) attempts to create upstream from the ingress-controller and this error:

creating upstream service-a.staging.443.svc
creating upstream service-b.staging.443.svc
time="2023-06-23T19:28:30Z" level=error msg="could not update kong admin" error="10 errors occurred:\n\twhile processing event: {Create} upstream service-a.staging.443.svc failed: HTTP status 409 (message: \"UNIQUE violation detected on '{name=\\\"service-a.staging.443.svc\\\"}'\")
...

And recreating the Ingress/KongIngress didn’t create the Kong service/route either, so I am a bit puzzled here.

6

@aj99 - looks like you have an issue with all the service/routes/upstream within the corresponding workspace. Can you try to re-create the corresponding workspace as well ?

FYI, I too faced this issue with 2.x but since the day I upgraded to 3.x, there is no such issue occurred yet for me.

7

@Sachin_Ghumbre, What do you mean by “workspace”?
Also, which version of Kong 3.x and Kong KIC version did you upgrade to to resolve this issue?
Does Kong v2.8.3 → v3.x upgrade fix this issue, or did get v2.x in a working state before upgrading to Kong v3.x?

8

Hi @aj99

If you are using Enterprise version of kong, you will see the workspace under which you have created kong configs such as routes/services/etc.

Since when I upgraded to 3.1.1 gateway, I did not see this error (till today).
Before upgrade, the old version 2.8.x had this issue for me. It used to come for the route and associated plugin. I just used to delete error specific plugin/route and redeploy the config and it used to work for me.

FYI, I did not upgrade to 3.1.1 just due to this error. I upgraded to the latest version (at the time of upgrade) due to our business requirement.

Even after your multiple tries, if you don’t find the fix, better you connect with Kong support team (Enterprise only).