Kong remove enabled plugin

Hey guys, hope you’re fine!
I’m using the following environment:
Kong 0.14.1
Kong Ingress Controller 0.2.0
My question is, there’s some bug with this versions when we talk about monitoring plugins, plugins like api-key and rate-limiting works fine, but when I enable the prometheus plugin, kong simple remove the plugin five minutes later, this occurs with statsd plugin too.
I used the oficial kong docs.
Help me please!
Thank you guys!!!

Hi,

Did you configure the plugin using a KongPlugin CRD? When using the Kubernetes Ingress Controller, you must configure plugins as such, since the controller will cleanup any manually added resources.

Hi, I used in that way now as your mention, see below:

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: prometheus-plugin
namespace: kong
labels:
global: “true”
disabled: false
config:
key: value
plugin: prometheus

And the output is the following:

curl -i http://localhost:8001/metrics

HTTP/1.1 200 OK
Server: openresty/1.13.6.2
Date: Tue, 11 Dec 2018 18:05:19 GMT
Content-Type: text/plain; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Access-Control-Allow-Origin: *

HELP kong_datastore_reachable Datastore reachable from Kong, 0 is unreachable

TYPE kong_datastore_reachable gauge

kong_datastore_reachable 1

HELP kong_nginx_http_current_connections Number of HTTP connections

TYPE kong_nginx_http_current_connections gauge

kong_nginx_http_current_connections{state=“accepted”} 1294
kong_nginx_http_current_connections{state=“active”} 1
kong_nginx_http_current_connections{state=“handled”} 1294
kong_nginx_http_current_connections{state=“reading”} 0
kong_nginx_http_current_connections{state=“total”} 1294
kong_nginx_http_current_connections{state=“waiting”} 0
kong_nginx_http_current_connections{state=“writing”} 1

HELP kong_nginx_metric_errors_total Number of nginx-lua-prometheus errors

TYPE kong_nginx_metric_errors_total counter

kong_nginx_metric_errors_total 0

Is that correct?
Because I can’t import this output in the official Kong dashboard ( https://grafana.com/dashboards/7424 ).

Thank you

Hello guys,
Could you help me please with this question?

Thank you

Hello @Eddie4Frost,

It seems like key:value under the config property in your YAML is invalid. Prometheus plugin doesn’t take any config with key as a “key”.

Please try the following:

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: prometheus-plugin
namespace: kong
labels:
global: “true”
plugin: prometheus

Also, please note that once you enable this plugin, you’ll need to configure Prometheus to scrape Kong nodes in order for these metrics to be available in Prometheus (which is what Grafana will read from).

Hey, I hope you are fine!
I’ve applied the code above and created the following scrape_config:

  • job_name: ‘kong-exporter’
    static_configs:
    - targets: [‘http://localhost:8001’]
    labels:
    group: ‘kong-exporter’

I’m still doing request with curl and I see the same metric listed above without success on Grafana too.

Thank you for all the support.

I’m assuming you’ve put this inside your Prometheus pod running in k8s.

This will not work because localhost:8001 is scraping the localhost of Prometheus pod.
You will need to configure Prometheus to use k8s service discovery to discover all Kong pods and scrape them.
Please refer the Prometheus documentation: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config

Hi there,
Thank you for your feedback @hbagdi, could you please take a look on these screenshots?

Kong now are sending metric to Prometheus, but only 3 metrics:
kong_datastore_reachable
kong_nginx_http_current_connections
kong_nginx_metric_errors_total

When doing a curl on metrics:

HELP kong_datastore_reachable Datastore reachable from Kong, 0 is unreachable

TYPE kong_datastore_reachable gauge

kong_datastore_reachable 1

HELP kong_nginx_http_current_connections Number of HTTP connections

TYPE kong_nginx_http_current_connections gauge

kong_nginx_http_current_connections{state=“accepted”} 36890
kong_nginx_http_current_connections{state=“active”} 3
kong_nginx_http_current_connections{state=“handled”} 36890
kong_nginx_http_current_connections{state=“reading”} 0
kong_nginx_http_current_connections{state=“total”} 84368
kong_nginx_http_current_connections{state=“waiting”} 2
kong_nginx_http_current_connections{state=“writing”} 1

HELP kong_nginx_metric_errors_total Number of nginx-lua-prometheus errors

TYPE kong_nginx_metric_errors_total counter

kong_nginx_metric_errors_total 0

The output is different from the official documentation. I don’t have any idea where is the issue.
Thank you again guys!

Good to see you making progress on this.

Can you make sure that you’ve created a KongPlugin resource for prometheus plugin in Kong?

Also, have you created an Ingress rules in your k8s cluster and is Kong actually proxying traffic?

The same problem to me.
When doing a curl on metrics:

HELP kong_datastore_reachable Datastore reachable from Kong, 0 is unreachable

TYPE kong_datastore_reachable gauge

kong_datastore_reachable 1

HELP kong_nginx_http_current_connections Number of HTTP connections

TYPE kong_nginx_http_current_connections gauge

kong_nginx_http_current_connections{state=“accepted”} 101406
kong_nginx_http_current_connections{state=“active”} 4
kong_nginx_http_current_connections{state=“handled”} 101406
kong_nginx_http_current_connections{state=“reading”} 0
kong_nginx_http_current_connections{state=“total”} 3320855
kong_nginx_http_current_connections{state=“waiting”} 3
kong_nginx_http_current_connections{state=“writing”} 1

HELP kong_nginx_metric_errors_total Number of nginx-lua-prometheus errors

TYPE kong_nginx_metric_errors_total counter

kong_nginx_metric_errors_total 0

There are only some info about nginx.

Kong version 0.14.
I have created the prometheus plugin in Kong and Kong proxyed traffic for period of time.

I delete the prometheus plugin for test.
I can still get the same info in /metrics uri.
It seem like prometheus is not working as expect.
Where is the data store of prometheus ?
Is there any way to find prometheus data or check prometheus is working well?