Before jumping into the question, I would like to give brief about our requirements (focused on configuration) of using Kong:
- We wanted to use Kong OSS on k8s with postgres as a DB which would be AWS RDS.
- We wanted to use most of the plugins at a service, route, consumer level with it’s own config customizations.
- We wanted to source-control the and use CI-CD to deploy/update Kong configurations.
Considering the above, We tried to install Kong using the all-in-one manifest for kong-with-postgres available on github.
- We deployed Kong Ingress controller in its own namespace which is up running and is using postgres running as a stateful-set.
- We decided to make an ingress of the application called “hello-world” to use kong and thus replaced ingressClassName to kong from nginx.
- We also created Rate-Limiting KongPlugin and added the annotation in ingress of hello-world application using kubectl command as shown in example in docs.
- After this, Since we had a requirement of source-controlling and using CI-CD, We came across Deck CLI tool which can be leveraged with CI-CD and installed the tool in local and tried to dump the configuration yaml file into local and found that the service for hello-world was added along with the route that we configured with rate-limiting was seen in the configuration file.
- The configuration file duplicates and adds default values to the plugin configuration for each route/service making the file larger in size.
This file has led us to many open questions which are listed below:
Do we need DecK to manage configurations on kubernetes as we actually run kubectl commands in-order to add kong annotations to the service/route or to make ingress to use kong ingress class.
Dumped kong.yaml configuration file doesn’t have the CRDs (Plugin) configuration in it. How to modify the plugin config in this case using DecK.
Do we actually need the configuration file (which will grow in size) to be stored and CI-CD to apply because we feel that adding annotations and modifying the CRDs on k8s using kubectl is more cloud native way of configuring things.
With all these above open questions in mind. We need your help in understanding what is the recommended/proper way to manage configurations of kong running on kubernetes with DB mode and how we can source-control the configurations and leverage CI-CD.
I am sure that most of the community members would have already tackled this situation.
Please suggest us on how to proceed on managing configurations.
Your inputs will be very much valuable to us.
Thanks and Happy Kong!