So you are reading into a Kong implementation of StatsD metrics as a plugin, which essentially sends Kong API metrics over UDP generally(can do TCP) to a remote host that accepts data in the statsD format, period delimited. Your next step in your Kong journey would be what application that supports StatsD formatted metrics should you store it in? Well I did some research and found a tool I really liked here for my implementation:
This tool will capture statsD metrics sent to it as well as visualize them for real-time charts , even format it for other data sinks such as Prometheus to scrape the data(now exposed from netdata) in a format it needs as well! Pretty cool huh? There are other tools out there to capture statsD metrics as well but I have not played with them any.
Thanks for all the input here. Meanwhile I understood that I need an additional tooling for consuming and viewing the collected metrics.
To make it more clear. I am looking for measurement of specific api accesses, storing it permanently, apply rate limiting and being able to make reports (don’t have to be nice, it is for billing on a monthly basis for some business-customers). It is not about live monitoring the running systems.
We already have here some grafana/prometheus in place (and some little knowledge in the team) for watching our kubernetes cluster.
I seems to be a good approach to try it with the prometheus/statsd_exporter. Simply for not having too much tooling. Also graphite seems to me a little bit complicated as we just need as an intermediate system for filling from there an other system and from what I read the underlying whisper system is not too good “out of the box” for permanent storage. I installed the docker-demo for the graphite system, but was not able to get the collected metrics:
Beside using statsd for measuring access to our apis we also need to store the original access log files (several months) for billing. In my understanding prometheus can also be used for long-time storage if configured with a long-time storage like described here https://prometheus.io/docs/prometheus/latest/storage/
There we could consume/store the statsd collected by kong plugin and either by an other kong plugin(?) all accesses to specific kong routes (where our services are mapped) or we use dropwizard metrics for doing it explicitely from our microservices.
We already have an authentication solution in place (and there are several others to come which are specific to the business-customers of our platform), so we simply want to route through kong without using this possibility of the kong api-gateway.
Any experience, critical feedback or opinions on this approach?
@gfestari I miss spoke a bit, I meant to phrase that netdata can support statsd recieved over TCP/UDP. Looks like Kong’s plugin is set to just transact over UDP, but I would check this file:
This can be modified to send as a TCP transaction certainly, I would fork the codebase and add a new schema value that lets you set udp or tcp and then the plugin can adapt. Not sure if Kong would like that as a PR to enable better capability. I know eventually they are looking to break the plugins out of the main codebase and into separate repo’s.
So quick update, I went for a bit of an alternative approach: using the HTTP Log Plugin to push service/routes logs to a small golang server which in turn exposes the metrics using the prometheus go library.
This is intended to be deployed as a sidecar container but I guess it could be run as a standalone install (I’m thinking similarly to the statsd-exporter approach)