Hello,
We run Kong in Kubernetes and a small percent of handled requests experience high x-kong-proxy-latency values: 300 - 6000 (some are even 12000), absolute majority of requests are <20ms.
What moving parts are included in kong proxy latency?
Searching around I’ve found that it is just kong’s processing of requests but what does it include in detail: database queries, DNS resolution, plugins, or something else?
In our case, we run a single plugin on most of the requests which is fairly simple and used to authorize access upstream. So on access, we are making a request to a separate app that either returns 401 or 200 allowing us to process the request further.
I’ve added metrics to this app and while it can be slow sometimes times of spike don’t match + it never reaches even 500ms not even mentioning 5000 or more.
Some requests with high proxy latency don’t even have that plugin configured (so public), yet their worst scenario time is less than 1000 as I see.
Can’t find any correlation with our Nodes usage, or spikes for this auth app. Scratching my head what else could be a reason for this behavior?
Update: I’ve found out that sending requests on slow client (rate-limited curl in my case) with big enough payload make x-kong-proxy-latency increase. This doesn’t explain all those hiccups though as some requests are simple GETs. Maybe only if clients connections were unstable for a moment…
Thank you.