# Getting 502 with increased average latency during performance testing **URL:** \ **Category:** General\ **Created:** [January 7, 2021, 2:23pm UTC](https://discuss.konghq.com/t/getting-502-with-increased-average-latency-during-performance-testing/7693 "2021-01-07T14:23:28Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Anuj\_Gupta](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/anuj_gupta/32/2473_2.png) [@Anuj\_Gupta](https://discuss.konghq.com/u/Anuj_Gupta)\ **Post date:** [January 7, 2021, 2:23pm UTC](https://discuss.konghq.com/t/getting-502-with-increased-average-latency-during-performance-testing/7693/1 "2021-01-07T14:23:28Z")
Hey, We have just one service and one route configured on kong in our load-test environment. API latency is generally **4ms** and we are able to achieve **1Million RPM** on 8cores and 16GB memory (C5.2xlarge ec2 AWS). At this RPM, the CPU reaches ~75%. On further increasing the load by **0.2M** , we start seeing requests failing with 502 status code. Average latency increases from **4ms to 300ms**. These requests with status code 502 don’t even reach the upstream server. I believe this is happening because of _“backlog”_ setting in _“kong.conf”_ file which is closing the connection when requests overflow the queue. But why does the latency increase? Out of 1.2M requests, we start getting somewhere around 400K requests with status code 502 and 800K requests with status code 200. This indicates these 502 requests are deteriorating the performance of 2XX requests also as load increases. **Ideal behavior we want to achieve:** Kong server should take as many requests as it can and serve them with 2XX status code. Any extra request which is too much to handle (CPU load breach, backlog queue breach, etc) should get dropped immediately without causing a rise in latency. **Configuration we are using:** Kong instances behind an AWS ALB limitnginx\_main\_worker\_rlimit\_nofile = auto nginx\_events\_worker\_connections = 2000 ulimit -n 100000 proxy\_listen = 0.0.0.0:8000 reuseport backlog=16384 We can see in AWS ALB logs status code 502 is returned and target-time is around 400ms. We have tried playing with various tuning parameters in _nginx.conf_ but haven’t been able to get better performance. Can someone please guide us in the right direction? I’ll add any more details if required. ---
**Author:** ![sivag](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/sivag/32/2944_2.png) [@sivag](https://discuss.konghq.com/u/sivag)\ **Post date:** [August 19, 2021, 10:37am UTC](https://discuss.konghq.com/t/getting-502-with-increased-average-latency-during-performance-testing/7693/2 "2021-08-19T10:37:09Z")
Hi @Anuj_Gupta , were you able to resolve the 502 errors? I am running into the same issue ---
**Author:** ![Anuj\_Gupta](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/anuj_gupta/32/2473_2.png) [@Anuj\_Gupta](https://discuss.konghq.com/u/Anuj_Gupta)\ **Post date:** [August 20, 2021, 11:59am UTC](https://discuss.konghq.com/t/getting-502-with-increased-average-latency-during-performance-testing/7693/3 "2021-08-20T11:59:55Z")
@sivag : Unfortunately no, we have not been able to find a workaround to this problem. We are exploring some ways to do load shedding in Kong by tracking in-flight requests, CPU and other parameters that can be of help. ---
**Author:** ![sivag](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/sivag/32/2944_2.png) [@sivag](https://discuss.konghq.com/u/sivag)\ **Post date:** [August 23, 2021, 3:40am UTC](https://discuss.konghq.com/t/getting-502-with-increased-average-latency-during-performance-testing/7693/4 "2021-08-23T03:40:59Z")
Thanks for the info @Anuj_Gupta