# Unexpected response searching a Kong Certificate: connect: connection refused **URL:** \ **Category:** Questions\ **Tags:** kubernetes\ **Created:** [March 26, 2019, 10:59am UTC](https://discuss.konghq.com/t/unexpected-response-searching-a-kong-certificate-connect-connection-refused/3110 "2019-03-26T10:59:53Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![bgarcial](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/bgarcial/32/776_2.png) [@bgarcial](https://discuss.konghq.com/u/bgarcial)\ **Post date:** [March 26, 2019, 10:59am UTC](https://discuss.konghq.com/t/unexpected-response-searching-a-kong-certificate-connect-connection-refused/3110/1 "2019-03-26T10:59:53Z")
Sometimes, my kong-ingress-controller pod is in `CrashLoopBackOff` status. ``` NAME READY STATUS RESTARTS AGE pod/kong-7f66b99bb5-747lm 1/1 Running 0 11d pod/kong-ingress-controller-7b6d8fff97-dqhqx 2/3 CrashLoopBackOff 649 5d2h pod/konga-85b66cffff-tkj8w 1/1 Running 0 11d ``` This happens with many frequency, and after to some minutes (usually more that 1 minute) my pod is running again. When this happen, I always check the logs and I get this `connect: connection refused` message: ``` al tcp 127.0.0.1:8001: connect: connection refused E0326 10:39:34.542954 6 controller.go:131] unexpected failure updating Kong configuration: Get http://localhost:8001/certificates/xxxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused W0326 10:39:34.542987 6 queue.go:113] requeuing kube-system/cert-manager-webhook, err Get http://localhost:8001/certificates/xxxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused I0326 10:39:37.875666 6 controller.go:128] syncing Ingress configuration... E0326 10:39:37.876260 6 kong.go:1142] Unexpected response searching a Kong Certificate: Get http://localhost:8001/certificates/xxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused E0326 10:39:37.876283 6 controller.go:131] unexpected failure updating Kong configuration: Get http://localhost:8001/certificates/xxxxx: dial tcp 127.0.0.1:8001: connect: connection refused W0326 10:39:37.876291 6 queue.go:113] requeuing kube-system/heapster, err Get http://localhost:8001/certificates/xxxxx: dial tcp 127.0.0.1:8001: connect: connection refused I0326 10:39:41.209755 6 controller.go:128] syncing Ingress configuration... E0326 10:39:41.210436 6 kong.go:1142] Unexpected response searching a Kong Certificate: Get http://localhost:8001/certificates/xxxxx: dial tcp 127.0.0.1:8001: connect: connection refused E0326 10:39:41.210449 6 controller.go:131] unexpected failure updating Kong configuration: Get http://localhost:8001/certificates/xxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused W0326 10:39:41.210456 6 queue.go:113] requeuing kube-system/tiller-deploy, err Get http://localhost:8001/certificates/xxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused I0326 10:39:44.542301 6 controller.go:128] syncing Ingress configuration... E0326 10:39:44.542809 6 kong.go:1142] Unexpected response searching a Kong Certificate: Get http://localhost:8001/certificates/xxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused E0326 10:39:44.542825 6 controller.go:131] unexpected failure updating Kong configuration: Get http://localhost:8001/certificates/xxxxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused W0326 10:39:44.542831 6 queue.go:113] requeuing kube-system/metrics-server, err Get http://localhost:8001/certificates/xxxxxxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused I0326 10:39:47.875673 6 controller.go:128] syncing Ingress configuration... E0326 10:39:47.876258 6 kong.go:1142] Unexpected response searching a Kong Certificate: Get http://localhost:8001/certificates/xxxx: dial tcp 127.0.0.1:8001: connect: connection refused E0326 10:39:47.876275 6 controller.go:131] unexpected failure updating Kong configuration: Get http://localhost:8001/certificates/xxxxxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused W0326 10:39:47.876299 6 queue.go:113] requeuing kong/kong-ingress-controller, err Get http://localhost:8001/certificates/xxxxxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused ``` I’ve performed the port-forward operation in order to check locally the behavior and I get: ``` ⟩ kubectl port-forward svc/kong-ingress-controller 8001:8001 -n kong Forwarding from 127.0.0.1:8001 -> 8001 Forwarding from [::1]:8001 -> 8001 Handling connection for 8001 E0326 11:46:44.589571 30810 portforward.go:400] an error occurred forwarding 8001 -> 8001: error forwarding port 8001 to pod 7ee8c115b05036b8a54046763d9aec1f9d69c8ed97b32a5406014ccb630484d2, uid : exit status 1: 2019/03/26 10:46:44 socat[5205] E connect(5, AF=2 127.0.0.1:8001, 16): Connection refused Handling connection for 8001 Handling connection for 8001 E0326 11:47:08.110395 30810 portforward.go:400] an error occurred forwarding 8001 -> 8001: error forwarding port 8001 to pod 7ee8c115b05036b8a54046763d9aec1f9d69c8ed97b32a5406014ccb630484d2, uid : exit status 1: 2019/03/26 10:47:08 socat[5473] E connect(5, AF=2 127.0.0.1:8001, 16): Connection refused E0326 11:47:08.110407 30810 portforward.go:400] an error occurred forwarding 8001 -> 8001: error forwarding port 8001 to pod 7ee8c115b05036b8a54046763d9aec1f9d69c8ed97b32a5406014ccb630484d2, uid : exit status 1: 2019/03/26 10:47:08 socat[5474] E connect(5, AF=2 127.0.0.1:8001, 16): Connection refused ``` And I make a curl operation and I get this. ``` ⟩ curl -i http://localhost:8001/ curl: (52) Empty reply from server ``` And I can’t use konga to connect to my kong-ingress-controller ![image](https://canada1.discourse-cdn.com/flex036/uploads/konghq/original/1X/513f3a3c941a45b38e08127063831afba1ace3e8.png) When my pod is running again, all it’s works again. I know the lifecycle pod dynamics, but why my `pod/kong-ingress-controller-7b6d8fff97-dqhqx` does not have a stable behavior. The `CrashLoopBackOff` status is very frequent in my kong deployment. Why happens this situation? ---
**Author:** ![hbagdi](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/hbagdi/32/562_2.png) [@hbagdi](https://discuss.konghq.com/u/hbagdi)\ **Post date:** [April 2, 2019, 6:04pm UTC](https://discuss.konghq.com/t/unexpected-response-searching-a-kong-certificate-connect-connection-refused/3110/2 "2019-04-02T18:04:51Z")
Do you have scheduling issues in you cluster? It seems like your pod is being terminated and restarted. Have you checked the logs of the `admin-api` container inside your Ingress Controller pod? It might have some useful information. ---
**Author:** ![bgarcial](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/bgarcial/32/776_2.png) [@bgarcial](https://discuss.konghq.com/u/bgarcial)\ **Post date:** [April 3, 2019, 10:05pm UTC](https://discuss.konghq.com/t/unexpected-response-searching-a-kong-certificate-connect-connection-refused/3110/3 "2019-04-03T22:05:40Z")
> Do you have scheduling issues in you cluster? What do you mean with scheduling issues This is the output of the `admin-api` logs ``` ⟩ kubectl logs pod/kong-ingress-controller-869bf4597c-8zrc6 -n kong -c admin-api 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /certificates/ee0c9a90-5652-11e9-884c-6a234afbbaf0 HTTP/1.1" 200 2874 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /snis/zcrm365dev.possibilit.nl HTTP/1.1" 200 163 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /certificates?size=1000 HTTP/1.1" 200 2897 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /upstreams/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 750 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /upstreams/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 750 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /upstreams/development.cm-acme-http-solver-27wrp.8089/targets?size=1000 HTTP/1.1" 200 193 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /upstreams/development.zcrm365-dev.80 HTTP/1.1" 200 734 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /upstreams/development.zcrm365-dev.80 HTTP/1.1" 200 734 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /upstreams/development.zcrm365-dev.80/targets?size=1000 HTTP/1.1" 200 193 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /consumers?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /services/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 316 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /services/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 316 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /services/7eff16b4-0b4d-4012-87ec-bf997f110e3a/plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /services/development.zcrm365-dev.80 HTTP/1.1" 200 284 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /services/development.zcrm365-dev.80 HTTP/1.1" 200 284 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /services/72025c72-588d-41fb-ae24-f2a869dc6b0a/plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /services?size=1000 HTTP/1.1" 200 624 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /routes?size=1000 HTTP/1.1" 200 824 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /services/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 316 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /routes?size=1000 HTTP/1.1" 200 824 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /routes/1cfcf831-68d3-470c-b8cd-10937acac14d/plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /services/development.zcrm365-dev.80 HTTP/1.1" 200 284 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /routes?size=1000 HTTP/1.1" 200 824 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /routes/aabe58de-4162-4f0d-8861-9aec42039800/plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1" 10.244.2.1 - - [03/Apr/2019:21:59:31 +0000] "GET /status HTTP/1.1" 200 206 "-" "kube-probe/1.12" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /certificates/ee0c9a90-5652-11e9-884c-6a234afbbaf0 HTTP/1.1" 200 2874 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /snis/zcrm365dev.possibilit.nl HTTP/1.1" 200 163 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /certificates?size=1000 HTTP/1.1" 200 2897 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /upstreams/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 750 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /upstreams/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 750 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /upstreams/development.cm-acme-http-solver-27wrp.8089/targets?size=1000 HTTP/1.1" 200 193 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /upstreams/development.zcrm365-dev.80 HTTP/1.1" 200 734 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /upstreams/development.zcrm365-dev.80 HTTP/1.1" 200 734 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /upstreams/development.zcrm365-dev.80/targets?size=1000 HTTP/1.1" 200 193 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /consumers?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /services/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 316 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /services/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 316 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /services/7eff16b4-0b4d-4012-87ec-bf997f110e3a/plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /services/development.zcrm365-dev.80 HTTP/1.1" 200 284 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /services/development.zcrm365-dev.80 HTTP/1.1" 200 284 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /services/72025c72-588d-41fb-ae24-f2a869dc6b0a/plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /services?size=1000 HTTP/1.1" 200 624 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /routes?size=1000 HTTP/1.1" 200 824 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /services/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 316 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /routes?size=1000 HTTP/1.1" 200 824 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /routes/1cfcf831-68d3-470c-b8cd-10937acac14d/plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /services/development.zcrm365-dev.80 HTTP/1.1" 200 284 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /routes?size=1000 HTTP/1.1" 200 824 "-" "Go-http-client/1.1" 127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /routes/aabe58de-4162-4f0d-8861-9aec42039800/plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1" 10.244.2.1 - - [03/Apr/2019:21:59:41 +0000] "GET /status HTTP/1.1" 200 206 "-" "kube-probe/1.12" 10.244.2.1 - - [03/Apr/2019:21:59:47 +0000] "GET /status HTTP/1.1" 200 206 "-" "kube-probe/1.12" 10.244.2.1 - - [03/Apr/2019:21:59:53 +0000] "GET /status HTTP/1.1" 200 206 "-" "kube-probe/1.12" 10.244.2.1 - - [03/Apr/2019:21:59:57 +0000] "GET /status HTTP/1.1" 200 206 "-" "kube-probe/1.12" [I] ``` But my kong-ingress-controller is in CrashLoopBackOf of a repeated way, this make that I have problems when I want to create new routes with new ingress. ``` Every 2,0s: kubectl get all -n kong el-pug: Thu Apr 4 00:38:09 2019 NAME READY STATUS RESTARTS AGE pod/kong-6875478957-6b7xk 1/1 Running 0 56m pod/kong-ingress-controller-869bf4597c-8zrc6 2/3 CrashLoopBackOff 17 56m pod/kong-migrations-fn6pm 0/1 Completed 0 56m pod/konga-85b66cffff-d2fjm 1/1 Running 0 127m ``` ---
**Author:** ![hbagdi](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/hbagdi/32/562_2.png) [@hbagdi](https://discuss.konghq.com/u/hbagdi)\ **Post date:** [April 10, 2019, 5:19am UTC](https://discuss.konghq.com/t/unexpected-response-searching-a-kong-certificate-connect-connection-refused/3110/4 "2019-04-10T05:19:05Z")
Hello @bgarcial, Were you able to solve this problem? If this problem is reproducible, please open an [issue](https://github.com/kong/kubernetes-ingress-controller/issues) with detailed reproducible steps so that we can replicate and fix the underlying issue. Thanks! ---
**Author:** ![bgarcial](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/bgarcial/32/776_2.png) [@bgarcial](https://discuss.konghq.com/u/bgarcial)\ **Post date:** [April 10, 2019, 7:15am UTC](https://discuss.konghq.com/t/unexpected-response-searching-a-kong-certificate-connect-connection-refused/3110/5 "2019-04-10T07:15:27Z")
Hi @hbagdi My kong-ingress-controller look stable and running now but when I perform changes to the ingress resources which point to it, sometimes have a `CrashLoopBackOff` status and this happens with certain frequency. The changes performed in the ingress resources which point to it, often create and remove routes, certificates and validate orders. Nothing weird. I am not sure exactly about what are the causes of this behavior. Let me think about and execute some tests in order to clarify this behavior and write of a detailed way the issue that you advise me in case of being necessary ---
**Author:** ![bgarcial](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/bgarcial/32/776_2.png) [@bgarcial](https://discuss.konghq.com/u/bgarcial)\ **Post date:** [April 10, 2019, 10:15am UTC](https://discuss.konghq.com/t/unexpected-response-searching-a-kong-certificate-connect-connection-refused/3110/6 "2019-04-10T10:15:14Z")
> [@hbagdi](#): > > If this problem is reproducible, please open an [issue](https://github.com/kong/kubernetes-ingress-controller/issues) with detailed reproducible steps so that we can replicate and fix the underlying issue. Thanks! Hi @hbagdi. I have write an issue detailing the behavior and the environment of this situation. It is here: [Unexpected response searching a Kong Certificate: connect: connection refuse · Issue #267 · Kong/kubernetes-ingress-controller · GitHub](https://github.com/Kong/kubernetes-ingress-controller/issues/267) The kong-ingress-controller getting down continuously. ---
**Author:** ![bgarcial](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/bgarcial/32/776_2.png) [@bgarcial](https://discuss.konghq.com/u/bgarcial)\ **Post date:** [April 18, 2019, 9:23am UTC](https://discuss.konghq.com/t/unexpected-response-searching-a-kong-certificate-connect-connection-refused/3110/7 "2019-04-18T09:23:27Z")
I was using a kong YAML manifest customized, which I didn’t add it some specific configurations with relation to the change in the URL for Kong admin API server value ( `-kong-url` ) which it’s recommended to be setted to `http://localhost:8001` byt he author of one helper which I am using, it is: [https://github.com/ollystephens/acme-kong-kube-helper](https://github.com/ollystephens/acme-kong-kube-helper) And of course I did that, I’ve changed the port to 8001, but I had to change this port value in other parts of my script, not only in the `ingress-controller` deployment resource, also in the `ingress-controller` and `kong-proxy` Services, and the other resource, along of the YAML manifest script. And also I had to set the script in some parts to `HTTP` scheme and not `HTTPS` scheme, in `ingress-controller` deployment resource. So, my Kong YAML customized manifest finally has stayed of [this way](https://gist.github.com/bgarcial/c83a9b70e73a56658316e886b9728792). I am going to close the issue that I have created at github page.