Prometheus: need help understanding metrics

Questions
1

Hi all,

I’m using Kong 1.4 with Prometheus plugin and need help understanding the latency metrics of my services.

The plugin indicates that:

Latencies Histograms : Latency as measured at Kong:

  • Request : Total time taken by Kong and Upstream services to serve requests.
  • Kong : Time taken for Kong to route a request and run all configured plugins.
  • Upstream : Time taken by the Upstream service to respond to requests.

So I thought that the Request latency was the sum of the Kong and Upstream latencies but on my Grafana dashboard I’m seeing a great divergence on these latencies:

image
image438×705 49.7 KB

How can I find what’s causing the difference between the Upstream and Request times?

Thank you