Kong latency issues

Questions
1

Hi everyone,

I posted my question on the gitter Mashape Kong channel before I realise it was written “Channel deprecated”.
Please allow me to copy/paste my original message below.

Hi, I’m Ramzi and we’ve been using Kong happily for almost 2 years.
We’ve been having some latency issues recently.

We use Kong to protect access to several APIs. We have several Kong instances on 3 different geographical sites (A, B, C) to ensure availability and resiliency. The APIs are also hosted on 2 different geographical sites (A, B).

Traffic between the 3 Kong sites is load balanced by a GTM (Global Traffic Manager). Upstream requests are also load balanced by a GTM between API sites.

It works pretty well except we experience kong latency on one Kong site. Let’s say on sites A and B everything is fine (10 ms max) but on C kong latency goes up to 200 ms.

This does not seem related to one specific API. When the latency is high, it is for any API. So I don’t think it is caused by the upstream servers.

And on site C, the Kong instances show a lot of ‘upstream prematurely closed connection while reading response header from upstream’ error messages.

I suspect a network problem. But I was wondering why it has such an impact on the kong latency.
Oh and one last thing. The APIs reply pretty fast. No timeout issue I guess.

Any hint about a test or any step I could take to diagnose my problem would be very much appreciated.

Ramzi

2

you could do a tracert from one of the servers (A) to the endpoint (B) and see the entire routing…

assuming, it’s a network node degraded maybe clear DNS and try again so it can choose a faster path?

1 Like
3

@ramzioueslati A first step would be to inspect the Kong latency headers X-Kong-Proxy-Latency and X-Kong-Upstream-Latency, and get a general sense of where the latency is coming from (Kong itself, or the connection between Kong and your upstream). From there, you can either start profiling Kong/Nginx, or probing your network as @balexandre suggested. If Kong seems to be the culprit (high X-Kong-Proxy-Latency), bear in mind that the latency could also be caused by slow DNS resolution, or a high database latency as well (or any other sort of I/O done by Kong). Profiling will help answering those questions in that case.

Good luck

4

Thanks for replying @balexandre and @thibaultcha.

I did actually check the latency headers and it’s definitely coming from Kong (x-kong-proxy-latency).

I still suspect a network issue as there are many ‘upstream prematurely …’ error messages on the Kong servers where we see high x-kong-proxy-latency. But I will try your suggestions and let you know.