Kong's Prometheus plugin not returning all metrics

Installation/Setup
1

Hello,

I followed the documentation for the Prometheus Kong plugin and setup was a breeze, but Kong isn’t returning all the metric data. Comparing my metrics with the sample output in Kong’s Prometheus repository reveals my setup is missing a lot of metrics. As expected loading the Kong Grafana dashboard only shows 3-4 charts populated with data while all the others report no data.

jharris@jharris-XPS-15-9570:~/Repository/helm.kong$ curl http://192.168.99.103:32449/metrics
# HELP kong_datastore_reachable Datastore reachable from Kong, 0 is unreachable
# TYPE kong_datastore_reachable gauge
kong_datastore_reachable 1
# HELP kong_nginx_http_current_connections Number of HTTP connections
# TYPE kong_nginx_http_current_connections gauge
kong_nginx_http_current_connections{state="accepted"} 183
kong_nginx_http_current_connections{state="active"} 2
kong_nginx_http_current_connections{state="handled"} 183
kong_nginx_http_current_connections{state="reading"} 0
kong_nginx_http_current_connections{state="total"} 1153
kong_nginx_http_current_connections{state="waiting"} 1
kong_nginx_http_current_connections{state="writing"} 1
# HELP kong_nginx_metric_errors_total Number of nginx-lua-prometheus errors
# TYPE kong_nginx_metric_errors_total counter
kong_nginx_metric_errors_total 0

jharris@jharris-XPS-15-9570:~/Repository/helm.kong$ minikube service list
|---------------|-----------------------------------------------|--------------------------------|
|   NAMESPACE   |                     NAME                      |              URL               |
|---------------|-----------------------------------------------|--------------------------------|
| kong          | kong-kong-admin                               | http://192.168.99.103:32449    |
| kong          | kong-kong-proxy                               | http://192.168.99.103:30263    |
|               |                                               | http://192.168.99.103:30986    |
|---------------|-----------------------------------------------|--------------------------------|

Kong’s helm chart version: 0.17.0
Kong version: 1.3.0

jharris@jharris-XPS-15-9570:~/Repository/helm.kong$ kubectl logs kong-kong-57fc5b995d-cmlfd -n kong -c ingress-controller
-------------------------------------------------------------------------------
Kong Ingress controller
  Release:    0.5.0
  Build:      922ae26b
  Repository: git@github.com:Kong/kubernetes-ingress-controller.git
  Go:         go1.12.6
-------------------------------------------------------------------------------

I0923 17:58:16.533105       1 main.go:303] Creating API client for https://10.96.0.1:443
I0923 17:58:16.541224       1 main.go:347] Running in Kubernetes Cluster version v1.15 (v1.15.2) - git (clean) commit f6278300bebbb750328ac16ee6dd3aa7d3549568 - platform linux/amd64
I0923 17:58:16.658425       1 main.go:148] kong version: 1.3.0
I0923 17:58:16.658448       1 main.go:151] Kong datastore: off
I0923 17:58:16.778408       1 controller.go:235] starting Ingress controller
I0923 17:58:16.781372       1 status.go:199] new leader elected: kong-kong-57fc5b995d-lx68w
I0923 17:58:16.827690       1 controller.go:128] successfully synced configuration to Kong
I0923 17:58:53.758971       1 controller.go:128] successfully synced configuration to Kong
I0923 17:58:56.292814       1 controller.go:128] successfully synced configuration to Kong
I0923 17:58:59.614192       1 controller.go:128] successfully synced configuration to Kong

Here’s my helm chart config:

replicaCount: 3

postgresql:
  enabled: false

cassandra:
  enabled: false

ingressController:
  enabled: true
  replicaCount: 3

admin:
  useTLS: false
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: 8444

readinessProbe:
  httpGet:
    scheme: HTTP

livenessProbe:
  httpGet:
    scheme: HTTP

env:
  database: "off"

dblessConfig:
  config:
    _format_version: "1.1"
    plugins:
    - name: prometheus
2

You shouldn’t be using Ingress Controller and dbless configuration at the same time.
The DB-less configuration (dblessConfig) will be replaced by Kong Ingress Controller.

Please create a CRD KongPlugin with name and plugin set to Prometheus.

Have a look at the following guide:
https://github.com/Kong/kubernetes-ingress-controller/blob/master/docs/guides/prometheus-grafana.md

4

I followed the guide, but still can not see the metrics on prometheus

5

Do you see Kong listed in the targets page in prometheus?

6

Hi, I am also facing the same issue, I used ServiceMonitor initially which gave me list of matrices:
kong_memory_lua_shared_dict_bytes
kong_memory_lua_shared_dict_total_bytes
kong_datastore_reachable
kong_memory_workers_lua_vms_bytes
kong_nginx_http_current_connections
kong_nginx_metric_errors_total
kong_upstream_target_health

But after installing global plugin Prometheus plugin | Kong Docs, I did not get any extra matrices. Also added below annotations to pod:

prometheus.io/scrape: “true”
prometheus.io/port: 8100

But did not help.

What could be the reason?