Ingress Controller keeps syncing new configs even when the configuration doesn't change

Questions
1

Summary

My Ingress Controller is having a strange behaviour.

From time to time, the Proxy purges the cache:

2021/06/04 10:17:12 [notice] 22#0: *5917 [lua] init.lua:260: purge(): [DB cache] purging (local) cache, client: 127.0.0.1, server: kong_admin, request: "POST /config?check_hash=1 HTTP/1.1", host: "localhost:8444"
2021/06/04 10:17:12 [notice] 22#0: *5917 [lua] init.lua:260: purge(): [DB cache] purging (local) cache, client: 127.0.0.1, server: kong_admin, request: "POST /config?check_hash=1 HTTP/1.1", host: "localhost:8444"
127.0.0.1 - - [04/Jun/2021:10:17:12 +0000] "POST /config?check_hash=1 HTTP/1.1" 201 35612 "-" "Go-http-client/1.1"
2021/06/04 10:17:16 [notice] 22#0: *5986 [lua] init.lua:260: purge(): [DB cache] purging (local) cache, client: 127.0.0.1, server: kong_admin, request: "POST /config?check_hash=1 HTTP/1.1", host: "localhost:8444"
2021/06/04 10:17:16 [notice] 22#0: *5986 [lua] init.lua:260: purge(): [DB cache] purging (local) cache, client: 127.0.0.1, server: kong_admin, request: "POST /config?check_hash=1 HTTP/1.1", host: "localhost:8444"
127.0.0.1 - - [04/Jun/2021:10:17:16 +0000] "POST /config?check_hash=1 HTTP/1.1" 201 35677 "-" "Go-http-client/1.1"
2021/06/04 10:17:19 [notice] 22#0: *6020 [lua] init.lua:260: purge(): [DB cache] purging (local) cache, client: 127.0.0.1, server: kong_admin, request: "POST /config?check_hash=1 HTTP/1.1", host: "localhost:8444"
2021/06/04 10:17:19 [notice] 22#0: *6020 [lua] init.lua:260: purge(): [DB cache] purging (local) cache, client: 127.0.0.1, server: kong_admin, request: "POST /config?check_hash=1 HTTP/1.1", host: "localhost:8444"
127.0.0.1 - - [04/Jun/2021:10:17:19 +0000] "POST /config?check_hash=1 HTTP/1.1" 201 35603 "-" "Go-http-client/1.1"

This triggers some syncing in the Ingress Controller :

time="2021-06-04T10:17:12Z" level=info msg="syncing configuration" component=controller
time="2021-06-04T10:17:12Z" level=info msg="successfully synced configuration to kong" component=controller
time="2021-06-04T10:17:16Z" level=info msg="syncing configuration" component=controller
time="2021-06-04T10:17:16Z" level=info msg="successfully synced configuration to kong" component=controller
time="2021-06-04T10:17:19Z" level=info msg="syncing configuration" component=controller
time="2021-06-04T10:17:19Z" level=info msg="successfully synced configuration to kong" component=controller

The thing is I haven’t changed anything.

Sometimes the syncing completes without any errors (or it doesn’t sync at all, ie, level=info msg="no configuration change, skipping sync to kong" component=controller), so I can access my endpoint, sometimes the syncing is not correct, and I can’t access my endpoint so I get:

curl -H "Authorization: Bearer $TOKEN" https://foo.bar/baz
{
  "message":"You cannot consume this service"
}

Why could this be happening? What info can I provide to help debug this problem?

Thank you in advance!

Kong Ingress controller version: 1.2

Kong OSS: 2.4

Kubernetes version

Client Version: version.Info{Major:“1”, Minor:“18”, GitVersion:“v1.18.16”, GitCommit:“7a98bb2b7c9112935387825f2fce1b7d40b76236”, GitTreeState:“clean”, BuildDate:“2021-02-17T12:01:24Z”, GoVersion:“go1.13.15”, Compiler:“gc”, Platform:“linux/amd64”}

Server Version: version.Info{Major:“1”, Minor:“18”, GitVersion:“v1.18.16”, GitCommit:“7a98bb2b7c9112935387825f2fce1b7d40b76236”, GitTreeState:“clean”, BuildDate:“2021-02-17T11:52:32Z”, GoVersion:“go1.13.15”, Compiler:“gc”, Platform:“linux/amd64”}

2

Do you still see this issue with 1.3?

Fix/snis compare by jiachinzhao · Pull Request #1268 · Kong/kubernetes-ingress-controller · GitHub fixed one known cause of unnecessary config updates. Some generated resources were randomly-ordered prior to that change, and the diff check cares about that even though it doesn’t actually matter (the config objects are an un-ordered set, but the diff still takes order into account).

There may still be other resources with similar issues, so if you’re still seeing unnecessary config POSTs after, can you collect configuration dumps before and after a POST and diff the JSON blobs? If we can see what’s still changing and confirm that the change is unnecessary, we should be able to rework config generation for the changed objects to ensure they don’t result in unnecessary updates.

3

I’m still using 1.2 and I was able to fix this problem.

I had two different ACL configured on a Service. Each ACL allowed access from different Consumers.

I changed that to have only one ACL per Service and everything is working as expected.