Hello there,
I have Ingress controller 2.0.6 and gateway 2.6.0. I have around a thousand of ingress files. All routes(ingresses) has 2-3 plugins each. I add ingresses and kongplugins for the tenants that we have and the kong ingress controller create the routes and plugins into kong proxy.
I am having some issues where what I apply/delete in kubernetes doesn’t match what is generated in the ingress controller. Sometimes it doesn’t create all the routes in the proxy, sometimes the plugins doesn’t get created or even when I delete the ingress files from kubernetes, the routes doesn’t get deleted completely. As an example, I had 36 routes created in the proxy and then I deleted these routes using kubectl delete -f /path/to/ingresses and the ingress controller only deleted 12 routes instead of 36.
The only reasons we think it might happens is A) we might have too many ingress files or B) We had some pods evicted due to disk pressure on some k8s nodes but we restarted and there are no issues showing up.
Any idea what could cause the issues?
Edit: I did some other tests, creating again 36 routes, only 35 gets created. I delete again using kubectl delete -f and I see that 32 gets deleted on 35. Looks like the ingress-controller can’t keep up with all the files that I have.