Latency-Based Loadbalancing in Upstreams

Feature Suggestions
#1

Hi Kong Team,

Currently, LB to targets in an upstream employs a weighted-RR LB pattern.

It would be very beneficial for certain deployment configurations (particularly multi-dc) if we could support load balancing based on data-center or (preferably) healthcheck ping latency.

What I suggest: Is to add a new loadbalancing option to Kong for a node to locally cache the most recent healthcheck ping response times for all targets in an upstream, and route exclusively to the target with the lowest response times. This would allow users to greatly customize their routing, as it would largely be based on the frequency and concurrency of healthchecks.

I’d love some guidance from the Kong team on where to start looking to add some of this functionality!

1 Like
#2

yes that is absolutely an interesting feature to add. But not an easy one to implement.

There are multiple components involved in this whole chain.

  1. The healthcheck library would need to track latencies.
  2. Kong would need to use the latencies to update weights in the LB

Some things that require attention:

  • weight = 0 means don’t add to LB currently
  • keep track of targets added by name (a single name might add multiple IP addresses to the balancer, there is a related issue with the healthchecks currently)
  • how many datapoints to use for latency calculation? incidental latency spikes might have a big impact otherwise
1 Like
#3

Thanks! Those are some very solid pointers. I’ll see what I can do here - I’d love to ultimately PR this

#4

Via active health checks:

  1. Take the avg latency of a configurable seen last X latency checks of the targets to dictate where to route. Until the latency aware routing can kick in(after # of checks) can route to various healthy targets. Or route to the first lowest latency target seen? Whatever makes sense there is fine.

The req is that the targets are healthy ofc too, we don’t care to route to a super fast target throwing Errors back at us on the health checks.