Kong kubernetes ingress controller failed to connect to admin

Hi, I have just tried kong ingress controller and got stuck with a weird issue, below are my steps:

  • Create a k8s cluster using kuberspray
  • Create PV for postgresql and run “helm install stable/kong --set ingressController.enabled=true -n kong --namespace kong”

Log from kong controller:

P/s: Im using Ubuntu 16.04 for host machine and setup k8s cluster using 2 vm(s) with Ubuntu 16.04 too.

I got this error again, at the first setup everything worked well but after reboot VMs the error occurred like magic.

Ah!
I have been struggling with this exact issue all day. I am using the rancher docker containers though. I have tried with both 0.8.0 and 0.9.0 versions of the kong-ingress-controller in conjunction with kong 1.5.0. This is the first time i have had this issue, and we have about 6 other clusters, so i assumed it was something with my environment - likely dns, possibly the ubuntu.

Do you have any further insight into this issue @giniz? I am pretty well confused. I am using ubuntu 18.04.

@thatbenguy Can you please share the logs of the controller and proxy containers in the ingress-kong pod?

Sure thing!

Note: I don’'t always see the last line “i/o timeout” error but it’s possible it just doesn’t get flushed in time before the container is restarted.

ingress-controller

k logs -f ota-kong-kong-767b64c974-b9rhh -c ingress-controller
-------------------------------------------------------------------------------
Kong Ingress controller
  Release:    0.8.0
  Build:      4b80e3f
  Repository: git@github.com:kong/kubernetes-ingress-controller.git
  Go:         go1.14.1
-------------------------------------------------------------------------------

W0608 19:45:45.252045       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0608 19:45:45.252214       1 main.go:442] Creating API client for https://10.43.0.1:443
I0608 19:45:45.257151       1 main.go:486] Running in Kubernetes Cluster version v1.17 (v1.17.6) - git (clean) commit d32e40e20d167e103faf894261614c5b45c44198 - platform linux/amd64
F0608 19:46:15.259351       1 main.go:185] making HTTP request: Get "https://localhost:8444/": dial tcp 91.195.241.136:8444: i/o timeout

proxy

k logs -f ota-kong-kong-767b64c974-b9rhh -c proxy
2020/06/08 19:42:32 [notice] 1#0: using the "epoll" event method
2020/06/08 19:42:32 [notice] 1#0: openresty/1.15.8.3
2020/06/08 19:42:32 [notice] 1#0: built by gcc 9.2.0 (Alpine 9.2.0) 
2020/06/08 19:42:32 [notice] 1#0: OS: Linux 5.3.0-53-generic
2020/06/08 19:42:32 [notice] 1#0: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2020/06/08 19:42:32 [notice] 1#0: start worker processes
2020/06/08 19:42:32 [notice] 1#0: start worker process 22
2020/06/08 19:42:32 [notice] 22#0: *1 [lua] cache_warmup.lua:46: cache_warmup_single_entity(): Preloading 'services' into the core_cache..., context: init_worker_by_lua*
2020/06/08 19:42:32 [notice] 22#0: *1 [lua] cache_warmup.lua:85: cache_warmup_single_entity(): finished preloading 'services' into the core_cache (in 0ms), context: init_worker_by_lua*
2020/06/08 19:42:32 [notice] 22#0: *1 [lua] cache_warmup.lua:46: cache_warmup_single_entity(): Preloading 'plugins' into the core_cache..., context: init_worker_by_lua*
2020/06/08 19:42:32 [notice] 22#0: *1 [lua] cache_warmup.lua:85: cache_warmup_single_entity(): finished preloading 'plugins' into the core_cache (in 0ms), context: init_worker_by_lua*

I also made another deployment with two containers in a single pod to test out just a simple curl of http://localhost and ensure that one container can access the other container in a single pod. This worked fine as expected, so i think my cluster is sane.

I also ran wget --no-check-certificate https://localhost:8444/ inside the proxy container and got a good response, so the admin api is operating as expected.

The code in the ingress controller is just grabbing the root api object.

// Root returns the response of GET request on root of
// Admin API (GET /).
func (c *Client) Root(ctx context.Context) (map[string]interface{}, error) {
	req, err := c.NewRequest("GET", "/", nil, nil)
	if err != nil {
		return nil, err
	}

	var root map[string]interface{}
	_, err = c.Do(ctx, req, &root)
	if err != nil {
		return nil, err
	}
	return root, nil
}

I am running out of ideas… I will probably modify and build the ingress-controller and re-deploy and see if i can’t learn anything.

Thanks for looking at this btw, @hbagdi

Can you change localhost to 127.0.0.1?