Latency increase at ~4000 RPS

Questions
1

Hi,

We are using Kong API gateway Open Source version 2.8.1 as for reverse proxy, caching, Rate limiting.
We are using it for real time ABR (adaptive bitrate streaming). Kong communicates with a upstream service which fetches data from other server. All setup is within a local network and single instance kong is running on a bare metal ubuntu linux 20.04. We are running 48 workers on a 64 cores machines.

We are facing issue when our request per second touches ~4000, latency starts shooting up drastically.
At 3750 RPS average latency is ~20 ms but post 4000 RPS average latency shoot to 3.75 secs.

Further points from debugging

  1. Delay is with kong, when in issue state if we directly request upstream it gives data without any delay.
  2. X-Kong-Proxy-Latency is very minimal
  3. While analysing netstat reports it suggest that kong TCP sockets are in ESTAB state but unable to read data as recv queue has large amount of data waiting to be read.
  4. X-Kong-Upstream-Latency is little high but primarily due to kong not reading socket data send by upstream.
  5. Issue is seen irrespective of HTTP or HTTPS.
  6. Upgrades to kong version 2.8.3 or latest 3.4.x didn’t helped.
  7. Reducing workers to 26 from 48 is helping a bit as we are able to reach 4.15K RPS but again after sometime latencies increases to average ~3.35 secs.

Please let us know how to debug this issue at kong level, any pointers are much appreciated.

Thanks