We’re running kong ingress controller, deployed typically via helm chart, with external postgresql DB and oauth2 plugin (thats why we couldn’t use dbless or hybrid mode) and muliple replicas on managed kubernetes by custom provider, which has k8s api server maintenance window every week and introduces up to 5min downtime of ingress controller.
The usual logs looks like:
ingress-controller E0913 22:14:38.672971 1 leaderelection.go:369] Failed to update lock: Put “https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kong-ingress-controller-leader-kong”: context deadline exceeded
ingress-controller I0913 22:14:38.673081 1 leaderelection.go:285] failed to renew lease kube-system/kong-ingress-controller-leader-kong: timed out waiting for the condition
…
could not stop DNS renewal for target removed from 81d2fda0-9a0f-4475-9a1e-3b15f9ae2b51: target object does not contain name and port, context: ngx.timer
recv() failed (111: Connection refused), context: ngx.timer
as well as:
[error] 1260#0: *5 [lua] events.lua:222: [events] could not broadcast crud event: failed to publish event: queue overflow, context: ngx.timer
[error] 1260#0: *5 [lua] worker.lua:170: failed to store event: queue overflow. data is :{“wid”:1,“event”:“mlcache:invalidations:kong_db_cache”,“data”:“oauth2_tokens:4209a7e2828fc69ebac3da0393822723a33a406b”,“source”:“mlcache”}, context: ngx.timer
Is there any possibility to disable API server check or increase the api call interval or whatever, than can make kong ingress still accept the requests (probably cache or buffer the configuration) during maintenance period? The k8s workers itself are running without any interruption.