Timeout error when accessing routes (apis)

Kubernetes
1

Hi all,

I am using the Kong Ingress Controller and kong databaseless to register routes using kubernetes resources on minikube.

When setting up ingress resources which link to a simple echo-service, I find that some requests to the kong gateway randomly timeout. The timeout occurs randomly, as sometimes the routes will respond as expected and sometimes they will fail.

I managed to gather these logs from the Kong container:

2019/07/23 00:40:11 [error] 36#0: *1049 [lua] balancer.lua:554: get_balancer(): balancer not found for test-service.default.svc, will create it, client: 172.17.0.1, server: kong, request: "GET /test-service?token=3333 HTTP/1.1", host: "..."
2019/07/23 00:40:41 [crit] 36#0: *1049 [lua] balancer.lua:615: on_upstream_event(): failed creating balancer for test-service.default.svc: timeout waiting for balancer for 6fe18e36-4728-40cf-9a08-3a666aadee21, client: 172.17.0.1, server: kong, request: "GET /test-service?token=3333 HTTP/1.1", host: "..."
2019/07/23 00:40:41 [error] 36#0: *1049 [lua] balancer.lua:584: on_target_event(): target create: balancer not found for test-service.default.svc, client: 172.17.0.1, server: kong, request: "GET /test-service?token=3333 HTTP/1.1", host: "..."
2019/07/23 00:41:11 [crit] 36#0: *1049 [lua] balancer.lua:615: on_upstream_event(): failed creating balancer for test-service.default.svc: timeout waiting for balancer for 6fe18e36-4728-40cf-9a08-3a666aadee21, client: 172.17.0.1, server: kong, request: "GET /test-service?token=3333 HTTP/1.1", host: "..."
2019/07/23 00:41:11 [error] 36#0: *1049 [lua] balancer.lua:584: on_target_event(): target create: balancer not found for test-service.default.svc, client: 172.17.0.1, server: kong, request: "GET /test-service?token=3333 HTTP/1.1", host: "..."
2019/07/23 00:41:41 [crit] 36#0: *1049 [lua] balancer.lua:615: on_upstream_event(): failed creating balancer for test-service.default.svc: timeout waiting for balancer for 6fe18e36-4728-40cf-9a08-3a666aadee21, client: 172.17.0.1, server: kong, request: "GET /test-service?token=3333 HTTP/1.1", host: "..."

I did see this topic on github that might be related, but I am using Kong v. 1.1.2, which includes the PR that fixes the linked issue.

Any help would be appreciated. Let me know if you need anything else.

Thanks,
Arnav

2

Few questions to dig into the root:

  • Do you have multiple pods of Kong Ingress running? If yes, is this issue specific to one pod or general across all?
  • Do you see any errors around POST /config access entries in the logs? We are looking to see if Kong doesn’t correctly create the load-balancer object.

If you can, please bump up to Kong 1.2.1, which contains a memory leak fix for this specific use-case.
Thanks!

2 Likes
3

It worked for me after choosing latest or 1.2.1 image :slight_smile:

4

Hi,

Sorry for the late response, I only just now got around to fixing this issue (since it was non-critical). Upgrading the version seems to have fixed the issue.

I think I do recall seeing some POST issues in the ingress-controllers logs. I’m not sure since it was a week ago.

Thanks for the help,
Arnav