Pros & Cons of KONG in Hybrid Mode

Hi all,

happy to get into touch with KONG and KONG nation, while just jumping into the challenge to getting KONG working.

The goal is to set up KONG as API Gateway as IAM solution, where the KONG application is containerized and being run scalable in the AWS and running the database on Premise elsewhere. The solution shall be legale to Privacy Shield, which means no data store within the AWS…

While the Hybride Mode - https://docs.konghq.com/2.0.x/hybrid-mode/ - sounds charming, this mode also has some limitations, like no dynamic token handling or OAuth support…

Therefore I’d like to get some feedback from other heads before jumping in the middle of it. For the KONG database Cassandra is being prefered, while KONGA could work as an admin GUI.

Looking forward for any good feedback and ideas according to this…

Thx
Sascha

1 Like

I too am just now digging into understanding the CP / DP (hybrid) architecture. Would love a deep dive with some Kong engineers on this pattern too if I ever get a chance to get some time with them. We are having lots of internal pattern discussions amongst our team and I am on the fence which direction to go.

As for the limitations, you can write an OAuth2.0 plugin that can leverage an external IDP to act as your OAuth2.0 server(It knows the client_id/client_secrets and distributes tokens and Kong just interfaces with that and caches results from it locally). Wouldn’t be too bad a plugin to write, I think Kong enterprise also offers a feature like that as well OOTB if you don’t wanna eat that dev cost.

As for using C* that seems fine as the DB of choice your CP Kong node would interface with. And Konga UI I imagine will interface with your CP Kong node too. Besides the OAuth2.0 work you will have to do, I think CP/DP model works for your use case to keep the DB and data on Prem and letting the CP push updates out to the DP Kong node off prem connected secured over MTLS.

My concerns with the DP and CP model are a few fold in complexity where you have multiple public clouds as well as your own datacenters and on prem infrastructure too, I am a novice really to public cloud work in terms of standing crap up in each big co: think AWS, Azure, Google Cloud are all big contenders and heavy hitters these days.

  1. It seems like a CP and DP model needs to have some kind of data segregation, where each DP node knows what proxies and routing config are specific to it. Reason being it makes sense that each DP node only has proxies specific to the APIs each DP Gateway node fronts in its respective public or private cloud(It doesn’t make sense for a Kong AWS DP node to route traffic to an Azure Microservice API you host, nor do I think it could ootb if not in the same network segment(again public cloud I am not an expert) ). So AWS Kong DP routes exclusive to AWS Microservices you host, Azure Kong DP routes exclusive to Azure Microservices you host and so forth with each DP getting its own exposed ingress gateway route. Seems the most performant design. So I view it as the proxy configs need to be name-spaced per Public cloud / On Prem Routing, or maybe as Kong is building out their “workspace” feature that is the full intent, to let each workspace be your proxies. I also think it makes sense to let the consumer Kong resources be a global of sorts though that live across all DP nodes, that way customers of your gateways regardless of calling on prem or a public cloud ingress endpoint gets the same experience already hooked in. Maybe that means just duplicating the consumer resource across each namespace/workspace if I want to achieve that.

I think all that workspacing stuff is an enterprise only feature(although I do notice now the community Kong instances are getting rewritten to leverage a default one, we are just missing the admin API management of em haha) which means in my case trying to leverage open source I would actually have to stand up N amount(N being # of Pub/Private cloud environments deploying to) of CP DP nodes for each of my Gateway routings which also would be a pain and turns me off from trying hybrid open source as its extra infra work and orchestration on my end to manage all those instances. Might as well just stand up standard independent DB-Less Kong nodes and push a yaml file out to em at that point if location of the original data store does not matter for me(which it likely doesn’t). That would save me 1/2 the infra nodes not needing that CP node and DB to store the persistant configs (could store each DP yaml in some other independent file storage and push it out myself and restart Kong with custom orchestration).

I think the strong points of the CP/DP model in a perfect world with Kong is the fact you get a standardized consumer experience. Error codes will always be the same, standard patterns for troubleshooting and auth/metrics/loggings. Familiarity with Kong means not having to learn a whole lot and writing a nice self-service interface that eventually programmatically works with the CP Kong node to produce changes would be super cool and an easy design. Such a pattern looks like this in my mind:

Some internally on the team are thinking a different design though where:

  1. For each public cloud you just leverage that public clouds API Gateway Offering(so google cloud its gonna be APIGee, Azure has their own Gateway and so does AWS), then on prem use w/e you have stood up(maybe its Kong) and go from there. Then leverage the ecosystem found in each public cloud/private cloud gateway offering. So far in some brief googling seems most Pub cloud offerings mostly offer config via API’s as one means if you so choose to interface with them programmatically. Not fully sure if the ecosystem is more involved than that from a customer experience as a Gateway owner, need to dig deeper there and study them.

My thoughts on weaknesses here are well you now have to learn how each of those pub cloud gateways work and support them, can you standardize their response behavior so you can have a good customer experience that feels integrated between all your clouds? Metrics? Troubleshooting? Logs? It’s an interesting discussion I would love to have with some folks that have ran into trying to solve this before at large scale. I personally don’t really see multi-cloud gateways and api’s as some crazy awesome thing (LT costs I am guessing for public cloud will go up as they know companies won’t wanna put the effort to build back on prem and manage their own infra, time will tell if that plays out 10+ years or more down the road but at scale these big cloud providers can secure superior infra and maintenance better than any independent company can) but that is a design we are trying to plan around some it seems like.

Would love to hear some opinions of the superior Kong engineering minds on this as I am just dipping my toes into thinking of “bigger picture” architectures here. Just a newb to it all :laughing: .

3 Likes

Doesn’t that all go along with compability of the ingress controller?

While Kubernetes itself supports and maintains GCE and nginx controllers. (described in https://git.k8s.io/ingress-nginx/README.md) the KONG ingress controller is one among some others:

  • AKS Application Gateway Ingress Controller

  • Ambassador API Gateway is an Envoy based ingress controller

  • AppsCode Inc. based ingress controller Voyager

  • AWS ALB Ingress Controller enables ingress using the AWS Application Load Balancer

  • Contour - is an Envoy - based ingress controller provided and supported by VMware.

  • Citrix provides an Ingress Controller for its hardware (MPX), virtualized (VPX) and baremetal and deployments.

  • F5 Networks provides for the F5 BIG-IP Container Ingress Services for Kubernetes

  • Gloo is an open-source ingress controller based on Envoy which offers API Gateway functionality with enterprise support from

  • HAProxy Ingress is a highly customizable community-driven ingress controller for HAProxy.

  • Istio based ingress controller

  • NGINX, Inc. offers support and maintenance for the NGINX Ingress Controller for Kubernetes.

  • Skipper HTTP router and reverse proxy for service composition, including use cases like Kubernetes Ingress, designed as a library to build your custom proxy

  • Traefik is a fully featured ingress controller, secrets, http2, websocket), and it also comes with commercial support by Containous.

and last but not least:

the Kong Ingress Controller for Kubernetes.

What would interest me deeply and should be specified within the KONG Hybrid Mode explanation is when to use it and for what purpose and dig deeper into advantages and disadvantages.

This is a great discussion topic, and all comments in this thread include great insights!

Jeremy’s comment is on point as it focuses on the architecture design: deciding to choose or not to use Hybrid Mode is not as much a matter of features and limitations, but an architectural choice.

This is not me trying to sugar-coat the limitations: what we call Hybrid Mode in Kong is really an architecture where you have a clear separation of Control Plane and Data Plane concerns. The Control Plane takes care of configuration, from its interface (for humans or other machines) to its storage (database); and the Data Plane takes care of the requests in flight. Under this light, the restrictions on what DP nodes can do are not only understood, but desired: you don’t want configuration work happening on these nodes.

One thing to have in mind is that, at peak performance, even in traditional mode Kong does not use the database: you get Kong’s full performance when all needed objects to serve requests fit in its in-memory cache. DB-less mode (which powers Hybrid Mode’s DP nodes) is this observation taken to the extreme: “well, just pre-fill all objects into the memory cache and get rid of the database, then!”

Some of the advantages of this DB-less mode are immediate: peak performance without having to warm up the cache! no database migrations in DP nodes! The limitation is that DP requests cannot write to the “mock database” that the DB-less cache implements. Why? One main reason is that DP nodes are fully independent from each other, and that is key to their performance, scalability and simplicity of operation. Plus, doing otherwise would amount to implementing an ad-hoc distributed database inside Kong, which doesn’t sound like the best of ideas. :slight_smile: If you need an actual database attached to your DP nodes, then we already support that in traditional non-Hybrid mode with industry-proven Postgres and Cassandra. And even in Hybrid mode, some plugins may have limited DB needs which can be solved on a case-by-case basic. For example, the rate-limiting plugin has an option to hook to Redis. The Kong data plane as a whole still runs DB-less (no migrations in proxy nodes, etc.), but you can still get cluster-wide rate-limits coordinated by all DP nodes sharing Redis tables.

A specific advantage of Hybrid as opposed to traditional is that the Hybrid DP is more resilient to CP outages than the traditional mode is to DB outages. At peak performance, a traditional node will run off the cache and short DB outages might even run unnoticed, but if it tries to write to the database while it is down you’ll run into errors (and it also implies extra care with which nodes can or cannot issue writes during live migrations). In Hybrid mode, when the CP is unavailable, the DP keeps running off the memory cache, and it also persists a copy of its configuration on disk so that it survives restarts.

Having said all that, Hybrid Mode is still new compared to traditional mode, having been introduced in version 2.0 in Kong OSS and in 2.1 in Kong Enterprise. Similarly to when we introduced DB-less mode back in Kong 1.1, we’ve been building out the feature set mostly driven by user feedback. DB-less has been a big success among our users, it is a proven pattern. This has given us confidence to use it as the foundation for Hybrid Mode.

In my observations, the adoption for Hybrid among our OSS userbase so far has been less quick than that of DB-less and I find that completely understandable: it is, in a sense, an even greater architectural shift than going DB-less for green field installations – one needs to shift their mindset towards this whole “planes” stuff. However, Jeremy is on point when observing that Hybrid Mode does give you a more uniform Kong experience: from an Admin API standpoint, you manage you Kong cluster pretty much the same way you always did.

But this is not the only pro. If we consider what it would entail to go DB-less as opposed to Hybrid Mode, which Jeremy described very well:

That would save me 1/2 the infra nodes not needing that CP node and DB to store the persistant configs (could store each DP yaml in some other independent file storage and push it out myself and restart Kong with custom orchestration).

At this point, you are pretty much implementing your own Control Plane by hand, managing the storage of your configuration by yourself, implementing custom orchestration and monitoring by yourself to push out updates to the data plane and ensuring the data plane is up-to-date, taking care of persistent caching at the DP side, and handling major migrations (any changes in the format of Kong objects) by updating YAML files by yourself… these are all things that a Control Plane node can do for you. (Small correction on the “1/2 the infra nodes”: you don’t need one CP node per DP node; one CP node can handle many DP nodes – you can scale your CP cluster and DP cluster independently according to your configuration vs. proxying needs for scaling and availability – that’s another benefit for CP/DP separation: these needs are often different.)

In a Kubernetes environment, some of these concerns get moved around, as the Kong Ingress Controller will handle the config synchronization and config storage moves into the Kubernetes paradigm, but the configuration and maintenance of Kong entities itself in YAML would still be up to you. (And even though K8s+KIC solves many of the problems, of course, not everybody wants to run K8s.)

As for the data segregation question: as of Kong 2.1, we don’t have automatic support for that, every DP node gets an identical configuration, meaning they essentially work to an outside observer as if you had a traditional DB-enabled Kong cluster, except that in the “CP nodes” you’d disable the proxy listening ports, and in the “DP nodes” you disabled the Admin API ports. Perhaps thinking of it this way makes it easier to compare the Hybrid deployment option to traditional DB-enabled Kong deployment. In that sense, I suppose that if you were doing a traditional DB-based Kong deployment and supporting multiple public clouds with nodes running with different configurations of routes, the solution would be to have separate clusters (possibly doing auth using a plugin that supports an external identity provider, for example).

We are very eager to gather feedback though on how people want to use Hybrid mode, of course, and we’re putting significant dev resources into its evolution, so the points raised here are very welcome!

1 Like

Really appreciate your insight here, the advantages certainly do stick out from the traditional architecture in general, especially if trying to expand from a single on prem model of Kong to something much bigger in design that spans across many different deployment environments.

Didn’t mean to confuse here but I think I may have, I was referring to what it would take to achieve total data segregation with OSS Hybrid mode in current implementation where say I have 1 Kong DP node per public cloud instance and I want it to have totally independent data(proxies) from other Kong DP nodes in other public cloud instances since each public cloud likely gets their own APIs by whatever cloud an API Provider of choice likes best for example. Maybe write tooling that lets me globalize the consumers and their access credentials though to still create a seamless experience from the customer perspective across all independent DP nodes on each respective cloud.

In such a model I don’t think folks would want a replicated set of routes etc. on each Kong DP node in each independent public cloud but rather a way to potentially push independent DP configurations from a single CP which comes with the ability to segregate data and identify a DP node that data should associate with(which probably requires having multiple MTLS keys associated with each DP node for the CP to use and tying those into knowing which workspace to leverage with each DP node would be how I think I would try to achieve it with the current ongoing db design and refactoring work).

I think the existing model serves well for having high availability to ensure as long as 1 cloud provider is up that a Kong DP node is active on and the services are reachable to its DNS + Network then you get a multi-cloud HA which is certainly cool!

To add extra context I will give a little snippit example here:

Hybrid capability now:

AWS:

  1. Upstream A
  2. Kong DP

Google Cloud:

  1. Upstream B
  2. Kong DP

Azure Cloud:

  1. Upstream C
  2. Kong DP

On Prem

  1. Upstream D
  2. Kong DP
  3. Kong CP

In current design, if you wanted 1 CP to rule all those DP, the dataplanes must share their data and proxy resources, even if you wanted each DP capable of routing just to the relevant APIs it fronts for the most performant network paths, or even some cases maybe a DP can’t even reach the other upstreams from where they sit so a connection timeout or DNS failure would occur. That is what I was getting at. Currently if you wanted to do so with hybrid and control each DP w separate configs you would need:

AWS:

  1. Upstream A
  2. Kong DP

Google Cloud:

  1. Upstream B
  2. Kong DP

Azure Cloud:

  1. Upstream C
  2. Kong DP

On Prem

  1. Upstream D
  2. Kong DP
  3. Kong CP 1 Controls AWS DP
  4. Kong CP 2 Controls GC DP
  5. Kong CP 3 Controls Azure Cloud DP
  6. Kong CP 4 Controls on Prem DP

We even discussed recently on a call another concept of what if you enable API providers to stand up their own Gateway instances via a standard provisioning process vs a Centralized Gateway on each cloud to handle all the traffic. Is it overkill and how to ensure you can still enforce governance and such against that provisioning model for APIs being developed by teams(I figure that can be baked into the gateway as a service offering to have some governance thrown in they would not be able to disable easily). I generally think decentralization is good but I also don’t see it as a requirement for most use cases, tx volumes at many places just don’t tip the scales unless you are aws/netflix/fb/google etc.

Within https://docs.konghq.com/2.1.x/hybrid-mode/ readers could get the impress to start with Generating Certificate/Key Pair, what’s right, but that does’t work from there with kong hybrid gen_cert … A better documentation would guide the user directly from the start and tell him what to be done before…

Doesn’t it make sense to start from initial environment setup?

What is the biggest difference between Hybrid Mode and service mesh ? many similarities,How do I understand that?

I believe Kong reduced most of the service mesh logic out of Kong since they have Kuma which will be a superior service mesh offering by design.

But a control plane / data plane model can still enable the traditional “gateway mode” many are familiar working with, which helps enable north/south traffic and minimize firewalls needed and creates a centralized right of passage for api traffic.

Service mesh to me has always seemed like a bigger help in east west traffic where your “mesh” is just the gateway decentralized and run as local sidecar on each of the API applications and microservices and then you connect directly to ones you need access too and they connect directly to ones they need access to to create a distributed but standardized way of auth/metrics/monitorings/discoverability/tracing and so forth. Ofc mesh can do north/south traffic too, just gonna have lots of fw rules to let all these clients reach all these independent ip’s of hosted services. But I have never implemented a service mesh or seen its benefits in the wild so take what I say with a grain of salt :smile: .

What would you suggest, if we would like to operate KONG within an hybrid environment between the AWS and a single on premise server where user information should be stored.

It would be great if we could also use KONGA as a GUI for KONG. Whereas KONGA does not support Cassandra, we’d prefer to set up this IAM on premise server with Cassandra.

For this server a configuration like this:

  • Intel Xeon E5-2630v4
  • 64 GB DDR4 ECC Reg
  • 2x 480 GB Samsung SM863
  • 1 Gigabit/s for Datatransfer
    should be fine.

Within the AWS a containerized environment - based on Kubernetes seems to be a good idea.

While under …/2.1.x/kong-for-kubernetes/install/ it states:

Using a database for Kong for Kubernetes

If you are using a database, we recommend running Kong in the in-memory mode (also known as DB-less) inside Kubernetes as all of the configuration is stored in the Kubernetes control-plane. This setup simplifies Kong’s operations, so no need to worry about Database provisioning, backup, availability, security, etc. If you decide to use a database, we recommend that you run the database outside of Kubernetes. You can use a service like Amazon’s RDS or a similar managed Postgres service from your cloud-provider to automate database operations.

We do not recommend using Kong with Cassandra on Kubernetes deployments, as the features covered by Kong’s use of Cassandra are handled by other means in Kubernetes.

I now wonder what way to go and how this fits alltogether. I would be glad if someone could help with some suggestions or adding critical thoughts to our project idea.

Thank you very much in advance!


© 2019 Kong Inc.    Terms  •  Privacy  •  FAQ