Kong Gateway load balancer doesn't rebalance

Reading the Kong Gateway load balancing reference and it makes the statement

The targets will be automatically cleaned when there are 10x more inactive entries than active ones. Cleaning will involve rebuilding the balancer, and hence is more expensive than just adding a target entry.

If I understand that statement, then if I have 2 healthy targets and 25 unhealthy targets, the balancer should rebuild and remove the unhealthy targets?

I have the case where services restart and get a new IP address witch causes the the old target to go unhealthy and be replaced by a new target. After time, I end up with close to if not more than 100 unhealthy targets which causes kong to mark the upstream as unhealthy. Maybe I misunderstand the statement or have something misconfigured?

@hicksd33 I’m going to provide a response from a colleague. I hope this helps with your question.

unhealthy targets are never automatically removed. They can set hostnames, in which case the balancer will resolve it to IP addresses and then balance over them. It will then (based on hostname ttl) automatically refresh the records and update the underlying IP addresses in the balancer.
From the description it seems to me they manually add IP addresses, and then expect Kong to remove those. Kong won’t do that, since an unhealthy target may recover. So they would have to manually remove the IP addresses again (same way they added them in the first place).
They are probably running into the threshold setting on the balancer, which is a threshold for taking an entire balancer offline (unhealthy) if the specified % of the total weight in the balancer is unhealthy (a means to prevent cascading failures of backends in case of an overload).

If you are seeing conflicting documentation, could you please post a link to the page so we can reference it internally and clarify the behavior. Thank you

Answer makes sense. We’ve come up with a way to remove the targets that won’t become healthy again so we can stick with ip routing and not use dns.

Here’s a link where kong docs state the targets get cleaned. There was another one but can’t seem to find it.