Reference HostIP for KongPlugin Declarations

Kubernetes
1

KongPlugin CRD

Any plugin dependent on a DaemonSet needs to be able to reference the HostIP to be able to interface with the dependent service (Datadog Plugin is the primary use case for this requirement)

Datadog Agent no longer supports Deployments with attached Services and instead only offers a DaemonSet. The Kong Datadog Plugin host field will be required to point to the HostIP to send metrics to dogstatsd.

If there is an alternative working solution to being able to reference HostIP variables in the KongPlugin CRD I would be open to that as well.

2

This is something weā€™re aware of, but there are some issues that make it difficult to fix right now. We need some way to make the host IP accessible to Kong, and while the downward API can provide this, limitations on both our and Kubernetesā€™ end prevent us from getting it:

  • Although the downward API allows access through either files or environment variables, thereā€™s an apparent bug that prevents accessing the host IP (and several other things) through files: https://github.com/kubernetes/kubernetes/issues/64168
  • Itā€™s not possible to access environment variables directly through Lua (i.e. the plugin code): the underlying NGINX server needs to expose them in advance. Weā€™re working on changes that should allow us to handle these more effectively, so weā€™ll probably revisit this issue once that becomes available.
3

Thank you for the prompt reply.

I also requested that https://github.com/kubernetes/kubernetes/issues/74265 be revisited so that the burden does not necessarily have to be on the application.

4

This is a bit of a necro but it seems thereā€™s been no official resolution on this yet and this is the newest thread I see :slight_smile:

I was tipped off that kong Lua can access environment variables from the ā€˜init phaseā€™ aka before any configuration is loaded, as seen when loading ambient authority environment variables.

With this in mind, itā€™s possible for a plugin to load a specific environment variable ā€“ as long as the variable name is hardcoded.

For example, I made a copy of the datadog plugin and added this line at the very top of handler.lua:

local ENV_DD_AGENT_HOST = os.getenv 'KONG_DD_AGENT_HOST'

(KONG_ prefix added because Kongā€™s official helm chart has literally no support for non-prefixed environment variables, lmao)

then I can use that value to ā€˜fix upā€™ the plugin config within the handler functions:

local function log(premature, conf, message)
  [...]
  if conf.host == "from-environment" and ENV_DD_AGENT_HOST then
    conf.host = ENV_DD_AGENT_HOST
  end

This appeared to work for me and it probably makes sense to upstream similar logic into the real datadog plugin. The current plugin is effectively incompatible with a normal Datadog daemonset installation.

Other threads Iā€™ve found:

5

PR welcome! This has been a known issue and solution for a while but hasnā€™t been implemented.

6

I donā€™t know if Iā€™m fully aligned on how the configuration part should work :stuck_out_tongue:

Iā€™ve seen multiple envvar names - DOGSTATSD_HOST_IP, DD_AGENT_HOST, STATSD_HOST, DD_AGENT_SERVICE_HOST to name a few - and this pattern wants the plugin to have the ā€˜correctā€™ one hardcoded since we seemingly have to read the env before we receive the plugin config.

The other aspect is the Kong helm chart doesnā€™t seem to allow specifying any of those variables because it forces a KONG_ prefix. So would we stick to the proper envvar and add a toggle to the Kong helm chart to add the status.hostIP stuff, or make a nonstandard KONG_DOGSTATSD_ variable so the helm chart can be left alone?

Since the variable name would be hardcoded in the plugin I suppose it makes sense to teach the helm chart the variable tooā€¦

I could PR if alignment shows up, but for the moment Iā€™ve simply told my Datadog agents to scrape kongā€™s Prometheus plugin instead :slight_smile:

7

Variable naming can be settled in a PR review.
If you could send a PR to Kong to update the Datadog plugin code to read from environment variable, that would be a good start and rest of the pieces should fall in place relatively easily.

8

This PR should resolve this use case:

1 Like