Performance issues in kong-ingress with requests with high latency
We are facing issues when we receive request, with high workload, for some processes that demand high latency periods.
Our conf:
ELB => k8s Ingress => Kong Ingress Controller + Kong
kong:1.5
kong-ingress-controller:0.5.0
Our db is Cassandra 3.11
And our problem is related with those request that take long term between request/response.
kubectl get events -n kong
LAST SEEN TYPE REASON OBJECT MESSAGE
92s Warning Unhealthy pod/kong-ingress-controller-5b4c8c757c-g8svn Readiness probe failed: Get http://10.0.16.149:8001/status: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
62s Warning Unhealthy pod/kong-ingress-controller-5b4c8c757c-g8svn Liveness probe failed: Get http://10.0.16.149:8001/status: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
6m20s Normal Pulled pod/kong-ingress-controller-5b4c8c757c-rx4bg Container image “kong:1.5” already present on machine
6m20s Normal Created pod/kong-ingress-controller-5b4c8c757c-rx4bg Created container admin-api
6m19s Normal Started pod/kong-ingress-controller-5b4c8c757c-rx4bg Started container admin-api
102s Warning Unhealthy pod/kong-ingress-controller-5b4c8c757c-rx4bg Readiness probe failed: Get http://10.0.10.107:8001/status: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
82s Warning Unhealthy pod/kong-ingress-controller-5b4c8c757c-rx4bg Liveness probe failed: Get http://10.0.10.107:8001/status: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
6m22s Normal Killing pod/kong-ingress-controller-5b4c8c757c-rx4bg Container admin-api failed liveness probe, will be restarted
logging kong-admin container we fond this:
2020/10/08 09:38:32 [warn] 26#0: *361922 [lua] ring.lua:238: redistributeIndices(): [ringbalancer 9] redistributed indices, size=10000, dropped=10000, assigned=0, left unassigned=10000, client: 127.0.0.1, server: kong_admin, request: “DELETE /upstreams/512a94e5-dc7f-455c-b883-7e637507ca48/targets/aa9ee14d-82ad-47c2-88fe-4e6e9fd34822 HTTP/1.1”, host: “localhost:8444”
2020/10/08 09:38:32 [warn] 26#0: *362911 [lua] balancer.lua:242: callback(): [healthchecks] balancer kong-ingress-controller.kong.svc reported health status changed to UNHEALTHY, context: ngx.timer, client: 127.0.0.1, server: 0.0.0.0:8444
can we solve this changing any setting on Kong ingress controller or is it any bugs ? any advice for solving this issue ??