Prometheus metric when upstream goes unhealthy

Is there any way where I can add custom metric to Prometheus? I would like to have an alert when upstream is down.

This is not possible out of the box as of today.
You can take the existing Promethus plugin and modify it to include these metrics.
This would also be a nice addition to the plugin so please feel free to PR it as well!

Does EE vitals/statsd metrics include this metric?

Kong Vitals includes some visibility into latency of the upstream but I don’t think it has metrics around (un)healthiness of targets of Upstreams (yet).

Hi @hbagdi,

I’m working on this now. I’m initially trying to load prometheus as a custom plugin(my-prometheus), so that I can edit the code locally. Kong is deployed on Kubernetes. Here’s what I did:

  1. mounted prometheus lua modules at /opt/kong/plugins/my-prometheus/ using ConfigMap

  2. KONG_LUA_PACKAGE_PATH: “/opt/?.lua;;”
    KONG_PLUGINS: “my-prometheus,http-log,correlation-id”

server {
  server_name kong_prometheus_exporter;
  listen 0.0.0.0:9542;
  access_log off;
  location / {
    default_type text/plain;
      content_by_lua_block {
        local prometheus = require "kong.plugins.my-prometheus.exporter"
        prometheus:collect()
      }
  }

  location /nginx_status {
    internal;
    access_log off;
    stub_status;
  }
}

And here is the error I’m getting:

2019/10/30 08:52:28 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: correlation-id
2019/10/30 08:52:28 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: http-log
2019/10/30 08:52:28 [error] 1#0: [kong] exporter.lua:14 my-prometheus: ngx shared dict 'prometheus_metrics' not found
nginx: [error] [kong] exporter.lua:14 my-prometheus: ngx shared dict 'prometheus_metrics' not found
2019/10/30 08:52:28 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: my-prometheus
2019/10/30 08:52:28 [notice] 1#0: using the "epoll" event method


2019/10/30 09:55:37 [error] 31#0: *37386 [kong] exporter.lua:114 my-prometheus: plugin is not initialized, please make sure  'prometheus_metrics' shared dict is present in nginx template, client: 192.168.2.181, server: kong_prometheus_exporter, request: "GET /metrics HTTP/1.1", host: "kong-admin.msr-wealth.svc:9542"
2019/10/30 09:55:37 [error] 31#0: *37386 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/pdk/private/phases.lua:66: no phase in kong.ctx.core.phase
stack traceback:
coroutine 0:
	[C]: in function 'error'
	/usr/local/share/lua/5.1/kong/pdk/private/phases.lua:66: in function 'check_phase'
	/usr/local/share/lua/5.1/kong/pdk/response.lua:559: in function 'collect'
	content_by_lua(kong-custom.conf:97):3: in function <content_by_lua(kong-custom.conf:97):1>, client: 192.168.2.181, server: kong_prometheus_exporter, request: "GET /metrics HTTP/1.1", host: "kong-admin.msr-wealth.svc:9542"

UPDATE: When I updated with bundled
KONG_PLUGINS: “bundled,my-prometheus”
it worked, but I’m getting metrics from both, which I don’t want.

Also, please suggest on the metric part. I’m a Lua newbie and after searching the Kong code, I could code like this:

  -- target reachable?
  local balancer = require "kong.runloop.balancer"
  local upstreams_dict = balancer.get_all_upstreams()
  for name, id in pairs(upstreams_dict) do
    local health_info = balancer.get_upstream_health(id)
    for target, status in pairs(health_info) do
      print("status: " .. status)
      print("target: " .. target)
      if status == "HEALTHY" then
        metrics.target_healthy:set(1)
      else
        metrics.target_healthy:set(0)
        kong.log.err("target is unhealthy ", target)
      end
    end
  end

@hisham @vinicius.mignot, could you guys help out here?

Raised PR. Please review.

@hbagdi, @hisham @vinicius.mignot: Need your expertise here, please

Two issues

  1. Works only when used along with ‘bundled’ : KONG_PLUGINS: “bundled,my-prometheus”
  2. Even if I can see the metrics when specifying ‘bundled’, I do NOT see proxy/kong_latency metrics.
    I do see the newly added custom metric:
HELP kong_upstream_target_health Upstream target healthy?, 0 is UNHEALTHY
TYPE kong_upstream_target_health gauge
kong_upstream_target_health{target="<IPAddr>:7777"} 1
kong_upstream_target_health{target="<IPAddr>:8888"} 1

Kong Version: 1.2.0 / DBLess

prometheus and my-prometheus plugins might have a conflict because they both will register handlers for /metrics.


© 2019 Kong Inc.    Terms  •  Privacy  •  FAQ