I have setup kong ingress controller 0.9.0 and Kong 2.0.4 on EKS. I used the all-in-one-dbless template with customization (internal nlb, resource limits, etc)
Requests from Kong to upstream services are made over HTTPS. The setup is working as expected and am now running performance tests.
I have 2 Kong Pods and an upstream service with simple API (minimal compute operation).
Following resource limits applied to Kong proxy
resources: requests: cpu: "200m" memory: "500Mi" limits: cpu: "500m" memory: "500Mi"
HPA is configured for Kong to scale up on 60% of CPU usage (reduced from 80)
HPA is configured for upstream to scale up on 80% cpu
Tests are run using wrk with connection 40 - 100 and threads 20 - 80 for 10 mins duration
As we drive higher load on this setup, we do see HPA kicking in for upstream service as expected.
However as the load increases, we see liveness probes start failing on Kong proxy causing it to be restarted. This results in errors on client side.
Throughout this period Kong pod’s CPU and MEM usage remains under 25% and 40% respectively so no HPA triggered.
In Kong error logs I see 3 categories of errors
- peer closed connection in SSL handshake while SSL handshaking to upstream (~95%)
- balancer.lua:628: get_balancer(): balancer not found for (~2.5%)
- connect() failed (111: Connection refused) while connecting to upstream (~2.5%)
It seems like Kong PODs are facing some network or sockets IO issues. What are the performance tuning options available in Kong?
Appreciate any help to resolve this.