Timeout error when accessing routes (apis)

Hi all,

I am using the Kong Ingress Controller and kong databaseless to register routes using kubernetes resources on minikube.

When setting up ingress resources which link to a simple echo-service, I find that some requests to the kong gateway randomly timeout. The timeout occurs randomly, as sometimes the routes will respond as expected and sometimes they will fail.

I managed to gather these logs from the Kong container:

2019/07/23 00:40:11 [error] 36#0: *1049 [lua] balancer.lua:554: get_balancer(): balancer not found for test-service.default.svc, will create it, client: 172.17.0.1, server: kong, request: "GET /test-service?token=3333 HTTP/1.1", host: "..."
2019/07/23 00:40:41 [crit] 36#0: *1049 [lua] balancer.lua:615: on_upstream_event(): failed creating balancer for test-service.default.svc: timeout waiting for balancer for 6fe18e36-4728-40cf-9a08-3a666aadee21, client: 172.17.0.1, server: kong, request: "GET /test-service?token=3333 HTTP/1.1", host: "..."
2019/07/23 00:40:41 [error] 36#0: *1049 [lua] balancer.lua:584: on_target_event(): target create: balancer not found for test-service.default.svc, client: 172.17.0.1, server: kong, request: "GET /test-service?token=3333 HTTP/1.1", host: "..."
2019/07/23 00:41:11 [crit] 36#0: *1049 [lua] balancer.lua:615: on_upstream_event(): failed creating balancer for test-service.default.svc: timeout waiting for balancer for 6fe18e36-4728-40cf-9a08-3a666aadee21, client: 172.17.0.1, server: kong, request: "GET /test-service?token=3333 HTTP/1.1", host: "..."
2019/07/23 00:41:11 [error] 36#0: *1049 [lua] balancer.lua:584: on_target_event(): target create: balancer not found for test-service.default.svc, client: 172.17.0.1, server: kong, request: "GET /test-service?token=3333 HTTP/1.1", host: "..."
2019/07/23 00:41:41 [crit] 36#0: *1049 [lua] balancer.lua:615: on_upstream_event(): failed creating balancer for test-service.default.svc: timeout waiting for balancer for 6fe18e36-4728-40cf-9a08-3a666aadee21, client: 172.17.0.1, server: kong, request: "GET /test-service?token=3333 HTTP/1.1", host: "..."

I did see this topic on github that might be related, but I am using Kong v. 1.1.2, which includes the PR that fixes the linked issue.

Any help would be appreciated. Let me know if you need anything else.

Thanks,
Arnav

Few questions to dig into the root:

  • Do you have multiple pods of Kong Ingress running? If yes, is this issue specific to one pod or general across all?
  • Do you see any errors around POST /config access entries in the logs? We are looking to see if Kong doesn’t correctly create the load-balancer object.

If you can, please bump up to Kong 1.2.1, which contains a memory leak fix for this specific use-case.
Thanks!

2 Likes

It worked for me after choosing latest or 1.2.1 image :slight_smile:

Hi,

Sorry for the late response, I only just now got around to fixing this issue (since it was non-critical). Upgrading the version seems to have fixed the issue.

I think I do recall seeing some POST issues in the ingress-controllers logs. I’m not sure since it was a week ago.

Thanks for the help,
Arnav


© 2018 Kong Inc.    Terms  •  Privacy  •  FAQ