How to manage Kong on kubernetes (DB-mode) configurations

Kubernetes
#1

Hi All,

Before jumping into the question, I would like to give brief about our requirements (focused on configuration) of using Kong:

  1. We wanted to use Kong OSS on k8s with postgres as a DB which would be AWS RDS.
  2. We wanted to use most of the plugins at a service, route, consumer level with it’s own config customizations.
  3. We wanted to source-control the and use CI-CD to deploy/update Kong configurations.

Considering the above, We tried to install Kong using the all-in-one manifest for kong-with-postgres available on github.

  1. We deployed Kong Ingress controller in its own namespace which is up running and is using postgres running as a stateful-set.
  2. We decided to make an ingress of the application called “hello-world” to use kong and thus replaced ingressClassName to kong from nginx.
  3. We also created Rate-Limiting KongPlugin and added the annotation in ingress of hello-world application using kubectl command as shown in example in docs.
  4. After this, Since we had a requirement of source-controlling and using CI-CD, We came across Deck CLI tool which can be leveraged with CI-CD and installed the tool in local and tried to dump the configuration yaml file into local and found that the service for hello-world was added along with the route that we configured with rate-limiting was seen in the configuration file.
  5. The configuration file duplicates and adds default values to the plugin configuration for each route/service making the file larger in size.

This file has led us to many open questions which are listed below:

  1. Do we need DecK to manage configurations on kubernetes as we actually run kubectl commands in-order to add kong annotations to the service/route or to make ingress to use kong ingress class.

  2. Dumped kong.yaml configuration file doesn’t have the CRDs (Plugin) configuration in it. How to modify the plugin config in this case using DecK.

  3. Do we actually need the configuration file (which will grow in size) to be stored and CI-CD to apply because we feel that adding annotations and modifying the CRDs on k8s using kubectl is more cloud native way of configuring things.

With all these above open questions in mind. We need your help in understanding what is the recommended/proper way to manage configurations of kong running on kubernetes with DB mode and how we can source-control the configurations and leverage CI-CD.

I am sure that most of the community members would have already tackled this situation.
Please suggest us on how to proceed on managing configurations.
Your inputs will be very much valuable to us.

Thanks and Happy Kong!

#2

You’ll usually just want to interact with the Kubernetes resources and manage those through CI/CD. The ingress controller is essentially translating Kubernetes resources into Kong configuration and then running deck sync on your behalf. There’s no reason to run deck directly when using the controller.

#3

Hi Traines,

Thanks for your reply.

Does it essentially mean to have all application’s Ingress/Service yaml files also in git along with custom resource definitions of kong and manage them through CI-CD?
If so, There would be huge number of yaml files in git considering two per application (ingress and service yaml).
Will it be appropriate to use CI-CD with those huge number of files.
Basically we are still in a confusion about what is the right way to start managing the configurations along with source controlling them.
If you can explain bit more comprehensively, It will be helpful to us.

Thanks.

#4

Any update on this ? @traines

#5

File count generally shouldn’t be a problem. Any CI/CD system worth using should be able to scale to whatever amount of configuration you have, but if in doubt, consult your Ci/CD system vendor. We don’t endorse or recommend any particular CI/CD tool–the main point of the last post was that if you’re managing configuration for an ingress controller-managed Kong instance, you should manage the Kubernetes resources, not the deck state file or DB-less configuration file that the controller creates from them.

You can combine Ingresses and Services (and any other YAML manifest) into a single file by separating them with a --- line (see YAML Ain’t Markup Language (YAML™) revision 1.2.2), but that doesn’t change how they function, it’s just personal organization preference.

#6

We are using YAML manifests as you suggested and they are working fine. Thanks @traines