Can I configure Kong to sync between multiple nodes in Kubernetes? I use Cassandra to sync in non-K8s environment, but I’m wondering if there’s something that relies on K8s’ ‘etcd’ or some other K8s technology that can be used instead. Also, if there is such configuration, will plug-ins work? Thanks.
This is pretty much exactly what our ingress controller does! 
It’s available via a values.yaml toggle if you use our Helm chart, and we provide example plain manifests if you do not.
Kong configuration is not stored in etcd directly: you’ll store standard Kubernetes Ingress/Service/etc. resources there and the controller translates them into Kong configuration and loads that through the admin API. Plugins and other Kong configuration that don’t fit directly into standard Kubernetes resources are handled via CRDs, e.g. the KongPlugin resource for plugin configuration.
1 Like
@traines, thank you for your reply! What if I’m constantly adding and removing consumers and changing my ACLs. Is there a way by which my Kong replicas will sync? Please forgive me if my questions are somewhat silly, I’m very new to K8s
The controller continuously works to update your Kong configuration to match your Kubernetes resources. The exact mechanism for the sync depends on whether you use a database or not:
- In DB-less mode, the controller instances
POST /configto apply the latest configuration they’ve derived from your Kubernetes resources. - In DB-backed mode, a controller
POST /routesandPOST /plugins(and so on for other resources) to the admin API, and those changes persist to the database. Kong nodes watch for database changes and update their local in-memory configuration based on them.
Generally speaking, you won’t need to worry about the specifics–Kong nodes keep their configuration up to date regardless of which mode you use: while they do so in different ways, the end result (all nodes are kept up to date) is the same.
As the Kubernetes resources are already a persistent store of configuration, we typically recommend DB-less mode. There are some other considerations for whether to use DB-less or DB-backed mode, but you can start with DB-less mode and switch over to DB-backed mode if you need it (since the configuration is ultimately sourced from Kubernetes resources, you can switch modes and the controller will translate your configuration to the new format automatically).
@traines, thank you again for your reply. One more question: if I do use the database (Cassandra / Scylla), how should i interpret “We do not recommend using Kong with Cassandra on Kubernetes deployments, as the features covered by Kong’s use of Cassandra are handled by other means in Kubernetes” at https://docs.konghq.com/2.0.x/kong-for-kubernetes/install/ ? Should i be deploying the database inside K8s or outside? Thank you in advance.
The typical use case for Cassandra is to persist your configuration across multiple active database nodes. When using Kong for Kubernetes, etcd provides similar functionality, although it’s storing the Kubernetes resource version of that configuration, which is then translated into Kong configuration by the controller.
Because the controller can regenerate that Kong configuration at any time so long as the corresponding Kubernetes resources remain in etcd, the characteristics of the database backing Kong don’t matter as much, as it’s effectively providing an ephemeral view of configuration that lives elsewhere.
Cassandra typically requires more infrastructure and expertise to operate than Postgres or DB-less mode, and as the additional replication and failover functionality it can provide isn’t as relevant when configuration is stored in etcd, we typically recommend against it in favor of the either two options. We typically recommend DB-less because it’s the more lightweight of the other options, though there’s some functionality that requires a database and it’s not suitable for all scenarios. In contrast, there’s no Kong-level functionality that requires using Cassandra over Postgres–functionality that requires a database will work with either.
Finally, parts of the Kong code work differently depending on which database you use, so there can be database-specific bugs or performance differences. While we’ve worked through most of the bugs affecting ingress controller functionality already, there are still some known performance issues related to tags that affect the controller’s configuration, and are difficult to address due to limitations in Cassandra’s schema features. Because of this and the overlap with etcd, we officially deprecated support for controller-managed deployments using Cassandra as of controller 0.9, and will remove support in the future. “Remove” in that sense doesn’t necessarily mean disable, but we won’t necessarily work to correct bugs that affect controller-managed deployments with Cassandra. They may still be fixed as part of a more general fix for Kong, but we won’t guarantee that.
Controller-managed deployments can usually migrate between databases without modifying configuration: if you’re currently using Cassandra, you can reconfigure Kong to use Postgres or DB-less and the controller will re-write your configuration in the new format. The exception is any configuration you’ve added manually, though you can use decK to handle that instead.