Capture Kong logs and send them to InfluxDB

Hi there!

We are starting a new API project leveraging Kong as our API Gateway.
We need to send Kong’s log (requests logs, timing, etc…) events to our influxdb timeseries database.

So I guess our options are as follow:

  1. Use an existing 'influxdb" plugin (community base?): After googling a bit and digging some posts from here, I haven’t found a plugin that would do the trick.
  2. Use an http existing plugin that would send to http listener (influxdb?) the logs from Kong.
  3. Create our own plugin (and contribute to the community :slight_smile: ). I haven’t see much information or tutorial on how to manage the end to end lifecycle of a plugin so I may have missed something… ?

My question to the community is quite simple, what would you suggest? what is the best option?

Thanks a lot for your help!

Cheers,

Jean-Baptiste

Hi, Kong has a http-log plugin which can be used to post traces to a third party via http. However I don’t think you can use it because the endpoint that Influxdb provides for writing data requires it to be in Line Protocol format. The http-log plugin only supports JSON. So that discards option 2 on your list.

Regarding option 1, I found this community plugin:

I have personally not used it, so I encourage you to test-drive it well before deciding to adopt it. Hopefully it’ll match your needs, or can be adapted to do so.

If you decide to build your own custom plugin, the starting point is the Plugin Development Guide. For your case I would start with the http-log plugin and change it to work with influx-db. One note about this:

  • The version in the next branch is a more advanced version of the plugin, able to “group several requests together in a buffer and send them in a single http request”, to mimimize traffic.
  • On the master branch there is a simpler version of the plugin, which is not able to do the “grouping of requests”. You will likely want to use this version as a base, since the influx-db endpoint doesn’t seem to support groupings.