Dealing with the lack of Kubernetes in Kong metric labels

Hi all, I’ve been running Kong in my Kubernetes clusters for a year or so now and one thing I cannot figure out is how everyone else is dealing with the labels/tags/facets that Kong’s prometheus plugin puts on the exposed metrics. For context, I’m scraping the metrics into Datadog, not Prometheus itself, but it seems like Kong’s datadog plugin has the exact same issue. So here’s my concern expressed using Prometheus metrics, which are easier to copy/paste:

kong_bandwidth{service="my-namespace.my-service.pnum-80",route="my-namespace.my-service.00",type="egress"} 510
kong_bandwidth{service="my-namespace.my-service.pnum-80",route="my-namespace.my-service.00",type="ingress"} 5959
kong_http_status{service="my-namespace.my-service.pnum-80",route="my-namespace.my-service.00",code="200"} 10

In that example there are two common tags which indicate where the requests went:

  • service="my-namespace.my-service.pnum-80"
  • route="my-namespace.my-service.00"

:arrow_right: My issue that these label values do not line up any other Kubernetes metrics related to a service. So if a team has a dashboard that breaks down traffic in their namespace by service name, or a dropdown to select a service to focus on, Kong’s HTTP metrics are not readily available to add to the dash! Maybe the strings can be manipulated at query-time but it’s not ideal and also not really possible in Datadog.

A much more useful metric sample could look like this, for example:

kong_bandwidth{namespace="my-namespace",service="my-service",port="80",type="egress"} 510
kong_bandwidth{namespace="my-namespace",service="my-service",port="80",type="ingress"} 5959
kong_http_status{namespace="my-namespace",service="my-service",port="80",code="200"} 10

Additionally, the target health metrics are pretty tough to make use of without a pod name:

# actual
kong_upstream_target_health{upstream="my-namespace.my-service.80.svc",target="10.60.0.181:80",address="10.60.0.181:80",state="healthy",subsystem="http"} 1
# ideal
kong_upstream_target_health{namespace="my-namespace",service="my-service",port="80",pod="my-service-5467d5c57b-5gvdb",state="healthy",subsystem="http"} 1

I understand that the crappy names for things are related to how Kong Ingress Controller is merely configuring Kong to behave in Kubernetes, instead of Kong itself being Kubernetes-aware. This was discussed previously: Service name and route name in grafana dashboard using prometheus plugin and ingress kind - #3 by antdem

:arrow_right: My question is how people deal work with these labels/tags. Do you always statically configure the Kong specific labels into any graph using these metrics? Do you dynamically handle the differences at query-time somehow? Do you not really correlate Kong metrics with your workload’s other metrics? Is there a fix I don’t know about?

Given Datadog’s inability to do string manipulation at query time, I’m looking at manipulating the Prometheus plugin’s responses to replace the labels with better ones. But if there’s better solutions I’d love to hear them.

Thanks for reading :slight_smile:

2 Likes