Statsd for measuring transactions - how to access the metrics

Hello,

I want to use kong for measuring transactions and the amount of users accessing our apis.

I tried locally currently to bring statsd to work.

Therefore setup locally kong by using docker with the example-service and the postgres db.


This works.

Now I tried to setup the stasd plugin for the example-service. This seems to work as described here:
https://getkong.org/plugins/statsd/

Here I can see the statsd plugin:
http://localhost:8001/plugins/

In my understanding (maybe wrong) here I should see the plugin
http://127.0.0.1:8001/apis/example-service/plugins

But this answers
{
“message”: “Not found”
}

What I don’t understand based on this documentation: where and how can I access the collected metrics?
kong.<api_name>.user.uniques
kong.<api_name>.request.count

Is there an Endpoint to access this?
Or do I have to look at the postgres DB?

I looked at the postgres db, but could not find anything.

Any hint?

Thanks
Matthias

Hello Matthias!

So you are reading into a Kong implementation of StatsD metrics as a plugin, which essentially sends Kong API metrics over UDP generally(can do TCP) to a remote host that accepts data in the statsD format, period delimited. Your next step in your Kong journey would be what application that supports StatsD formatted metrics should you store it in? Well I did some research and found a tool I really liked here for my implementation:

This tool will capture statsD metrics sent to it as well as visualize them for real-time charts :slight_smile: , even format it for other data sinks such as Prometheus to scrape the data(now exposed from netdata) in a format it needs as well! Pretty cool huh? There are other tools out there to capture statsD metrics as well but I have not played with them any.

Hope this helps.
-Jeremy

I have successfully used Graphite with Statsd as well. statsD -> Graphite -> Grafana

You can configure the StatsD Plugin for Kong to send data to https://github.com/prometheus/statsd_exporter , which in turn can be scraped by Prometheus (which in turn works well with Grafana for visualization)

Thanks for all the input here. Meanwhile I understood that I need an additional tooling for consuming and viewing the collected metrics.

To make it more clear. I am looking for measurement of specific api accesses, storing it permanently, apply rate limiting and being able to make reports (don’t have to be nice, it is for billing on a monthly basis for some business-customers). It is not about live monitoring the running systems.

We already have here some grafana/prometheus in place (and some little knowledge in the team) for watching our kubernetes cluster.

I seems to be a good approach to try it with the prometheus/statsd_exporter. Simply for not having too much tooling. Also graphite seems to me a little bit complicated as we just need as an intermediate system for filling from there an other system and from what I read the underlying whisper system is not too good “out of the box” for permanent storage. I installed the docker-demo for the graphite system, but was not able to get the collected metrics:
kong.<api_name>.user.uniques
kong.<api_name>.request.count

Beside using statsd for measuring access to our apis we also need to store the original access log files (several months) for billing. In my understanding prometheus can also be used for long-time storage if configured with a long-time storage like described here https://prometheus.io/docs/prometheus/latest/storage/
There we could consume/store the statsd collected by kong plugin and either by an other kong plugin(?) all accesses to specific kong routes (where our services are mapped) or we use dropwizard metrics for doing it explicitely from our microservices.
We already have an authentication solution in place (and there are several others to come which are specific to the business-customers of our platform), so we simply want to route through kong without using this possibility of the kong api-gateway.

Any experience, critical feedback or opinions on this approach?

@jeremyjpj0916 You mentioned the kong statsd plugin can do TCP. Went through the docs and code for the plugin and couldn’t find anything about this feature.

I know the statsd protocol supports TCP along with UDP, I’m asking because UDP’s somewhat restricted on my infrastructure/network.

Sorry for hijacking the thread and thanks!

@gfestari I miss spoke a bit, I meant to phrase that netdata can support statsd recieved over TCP/UDP. Looks like Kong’s plugin is set to just transact over UDP, but I would check this file:

This can be modified to send as a TCP transaction certainly, I would fork the codebase and add a new schema value that lets you set udp or tcp and then the plugin can adapt. Not sure if Kong would like that as a PR to enable better capability. I know eventually they are looking to break the plugins out of the main codebase and into separate repo’s.

1 Like

thanks for this @jeremyjpj0916 will get to hacking that udp into a tcp call

1 Like

So quick update, I went for a bit of an alternative approach: using the HTTP Log Plugin to push service/routes logs to a small golang server which in turn exposes the metrics using the prometheus go library.

This is intended to be deployed as a sidecar container but I guess it could be run as a standalone install (I’m thinking similarly to the statsd-exporter approach)

Repo’s over here: https://github.com/cilindrox/kong-prometheus-exporter, PRs welcome!

Going to keep it updated as its usage evolves on some production scenarios

1 Like

@gfestari super neat! will look it over sometime!