Unexpected response searching a Kong Certificate: connect: connection refused

Kubernetes
1

Sometimes, my kong-ingress-controller pod is in CrashLoopBackOff status.

NAME                                           READY   STATUS             RESTARTS   AGE
pod/kong-7f66b99bb5-747lm                      1/1     Running            0          11d
pod/kong-ingress-controller-7b6d8fff97-dqhqx   2/3     CrashLoopBackOff   649        5d2h
pod/konga-85b66cffff-tkj8w                     1/1     Running            0          11d

This happens with many frequency, and after to some minutes (usually more that 1 minute) my pod is running again. When this happen, I always check the logs and I get this connect: connection refused message:

al tcp 127.0.0.1:8001: connect: connection refused
E0326 10:39:34.542954       6 controller.go:131] unexpected failure updating Kong configuration: 
Get http://localhost:8001/certificates/xxxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused
W0326 10:39:34.542987       6 queue.go:113] requeuing kube-system/cert-manager-webhook, err Get http://localhost:8001/certificates/xxxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused
I0326 10:39:37.875666       6 controller.go:128] syncing Ingress configuration...
E0326 10:39:37.876260       6 kong.go:1142] Unexpected response searching a Kong Certificate: Get http://localhost:8001/certificates/xxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused
E0326 10:39:37.876283       6 controller.go:131] unexpected failure updating Kong configuration: 
Get http://localhost:8001/certificates/xxxxx: dial tcp 127.0.0.1:8001: connect: connection refused
W0326 10:39:37.876291       6 queue.go:113] requeuing kube-system/heapster, err Get http://localhost:8001/certificates/xxxxx: dial tcp 127.0.0.1:8001: connect: connection refused
I0326 10:39:41.209755       6 controller.go:128] syncing Ingress configuration...
E0326 10:39:41.210436       6 kong.go:1142] Unexpected response searching a Kong Certificate: Get http://localhost:8001/certificates/xxxxx: dial tcp 127.0.0.1:8001: connect: connection refused
E0326 10:39:41.210449       6 controller.go:131] unexpected failure updating Kong configuration: 
Get http://localhost:8001/certificates/xxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused
W0326 10:39:41.210456       6 queue.go:113] requeuing kube-system/tiller-deploy, err Get http://localhost:8001/certificates/xxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused
I0326 10:39:44.542301       6 controller.go:128] syncing Ingress configuration...
E0326 10:39:44.542809       6 kong.go:1142] Unexpected response searching a Kong Certificate: Get http://localhost:8001/certificates/xxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused
E0326 10:39:44.542825       6 controller.go:131] unexpected failure updating Kong configuration: 
Get http://localhost:8001/certificates/xxxxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused
W0326 10:39:44.542831       6 queue.go:113] requeuing kube-system/metrics-server, err Get http://localhost:8001/certificates/xxxxxxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused
I0326 10:39:47.875673       6 controller.go:128] syncing Ingress configuration...
E0326 10:39:47.876258       6 kong.go:1142] Unexpected response searching a Kong Certificate: Get http://localhost:8001/certificates/xxxx: dial tcp 127.0.0.1:8001: connect: connection refused
E0326 10:39:47.876275       6 controller.go:131] unexpected failure updating Kong configuration: 
Get http://localhost:8001/certificates/xxxxxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused
W0326 10:39:47.876299       6 queue.go:113] requeuing kong/kong-ingress-controller, err Get http://localhost:8001/certificates/xxxxxxxxx: dial tcp 127.0.0.1:8001: connect: connection refused

I’ve performed the port-forward operation in order to check locally the behavior and I get:

⟩ kubectl port-forward svc/kong-ingress-controller 8001:8001 -n kong
Forwarding from 127.0.0.1:8001 -> 8001
Forwarding from [::1]:8001 -> 8001
Handling connection for 8001
E0326 11:46:44.589571   30810 portforward.go:400] an error occurred forwarding 8001 -> 8001: error forwarding port 8001 to pod 7ee8c115b05036b8a54046763d9aec1f9d69c8ed97b32a5406014ccb630484d2, uid : exit status 1: 2019/03/26 10:46:44 socat[5205] E connect(5, AF=2 127.0.0.1:8001, 16): Connection refused
Handling connection for 8001
Handling connection for 8001
E0326 11:47:08.110395   30810 portforward.go:400] an error occurred forwarding 8001 -> 8001: error forwarding port 8001 to pod 7ee8c115b05036b8a54046763d9aec1f9d69c8ed97b32a5406014ccb630484d2, uid : exit status 1: 2019/03/26 10:47:08 socat[5473] E connect(5, AF=2 127.0.0.1:8001, 16): Connection refused
E0326 11:47:08.110407   30810 portforward.go:400] an error occurred forwarding 8001 -> 8001: error forwarding port 8001 to pod 7ee8c115b05036b8a54046763d9aec1f9d69c8ed97b32a5406014ccb630484d2, uid : exit status 1: 2019/03/26 10:47:08 socat[5474] E connect(5, AF=2 127.0.0.1:8001, 16): Connection refused

And I make a curl operation and I get this.

⟩ curl -i http://localhost:8001/
curl: (52) Empty reply from server

And I can’t use konga to connect to my kong-ingress-controller

image
image.png1353×406 37.4 KB

When my pod is running again, all it’s works again.

I know the lifecycle pod dynamics, but why my pod/kong-ingress-controller-7b6d8fff97-dqhqx does not have a stable behavior.

The CrashLoopBackOff status is very frequent in my kong deployment.
Why happens this situation?

2

Do you have scheduling issues in you cluster?
It seems like your pod is being terminated and restarted.

Have you checked the logs of the admin-api container inside your Ingress Controller pod? It might have some useful information.

3

Do you have scheduling issues in you cluster?

What do you mean with scheduling issues

This is the output of the admin-api logs

⟩ kubectl logs pod/kong-ingress-controller-869bf4597c-8zrc6 -n kong -c admin-api
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /certificates/ee0c9a90-5652-11e9-884c-6a234afbbaf0 HTTP/1.1" 200 2874 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /snis/zcrm365dev.possibilit.nl HTTP/1.1" 200 163 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /certificates?size=1000 HTTP/1.1" 200 2897 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /upstreams/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 750 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /upstreams/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 750 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /upstreams/development.cm-acme-http-solver-27wrp.8089/targets?size=1000 HTTP/1.1" 200 193 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /upstreams/development.zcrm365-dev.80 HTTP/1.1" 200 734 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /upstreams/development.zcrm365-dev.80 HTTP/1.1" 200 734 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /upstreams/development.zcrm365-dev.80/targets?size=1000 HTTP/1.1" 200 193 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /consumers?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /services/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 316 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /services/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 316 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /services/7eff16b4-0b4d-4012-87ec-bf997f110e3a/plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /services/development.zcrm365-dev.80 HTTP/1.1" 200 284 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /services/development.zcrm365-dev.80 HTTP/1.1" 200 284 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /services/72025c72-588d-41fb-ae24-f2a869dc6b0a/plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /services?size=1000 HTTP/1.1" 200 624 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /routes?size=1000 HTTP/1.1" 200 824 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /services/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 316 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /routes?size=1000 HTTP/1.1" 200 824 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /routes/1cfcf831-68d3-470c-b8cd-10937acac14d/plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /services/development.zcrm365-dev.80 HTTP/1.1" 200 284 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /routes?size=1000 HTTP/1.1" 200 824 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:29 +0000] "GET /routes/aabe58de-4162-4f0d-8861-9aec42039800/plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1"
10.244.2.1 - - [03/Apr/2019:21:59:31 +0000] "GET /status HTTP/1.1" 200 206 "-" "kube-probe/1.12"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /certificates/ee0c9a90-5652-11e9-884c-6a234afbbaf0 HTTP/1.1" 200 2874 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /snis/zcrm365dev.possibilit.nl HTTP/1.1" 200 163 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /certificates?size=1000 HTTP/1.1" 200 2897 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /upstreams/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 750 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /upstreams/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 750 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /upstreams/development.cm-acme-http-solver-27wrp.8089/targets?size=1000 HTTP/1.1" 200 193 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /upstreams/development.zcrm365-dev.80 HTTP/1.1" 200 734 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /upstreams/development.zcrm365-dev.80 HTTP/1.1" 200 734 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /upstreams/development.zcrm365-dev.80/targets?size=1000 HTTP/1.1" 200 193 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /consumers?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /services/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 316 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /services/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 316 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /services/7eff16b4-0b4d-4012-87ec-bf997f110e3a/plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /services/development.zcrm365-dev.80 HTTP/1.1" 200 284 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /services/development.zcrm365-dev.80 HTTP/1.1" 200 284 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /services/72025c72-588d-41fb-ae24-f2a869dc6b0a/plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /services?size=1000 HTTP/1.1" 200 624 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /routes?size=1000 HTTP/1.1" 200 824 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /services/development.cm-acme-http-solver-27wrp.8089 HTTP/1.1" 200 316 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /routes?size=1000 HTTP/1.1" 200 824 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /routes/1cfcf831-68d3-470c-b8cd-10937acac14d/plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /services/development.zcrm365-dev.80 HTTP/1.1" 200 284 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /routes?size=1000 HTTP/1.1" 200 824 "-" "Go-http-client/1.1"
127.0.0.1 - - [03/Apr/2019:21:59:32 +0000] "GET /routes/aabe58de-4162-4f0d-8861-9aec42039800/plugins?size=1000 HTTP/1.1" 200 23 "-" "Go-http-client/1.1"
10.244.2.1 - - [03/Apr/2019:21:59:41 +0000] "GET /status HTTP/1.1" 200 206 "-" "kube-probe/1.12"
10.244.2.1 - - [03/Apr/2019:21:59:47 +0000] "GET /status HTTP/1.1" 200 206 "-" "kube-probe/1.12"
10.244.2.1 - - [03/Apr/2019:21:59:53 +0000] "GET /status HTTP/1.1" 200 206 "-" "kube-probe/1.12"
10.244.2.1 - - [03/Apr/2019:21:59:57 +0000] "GET /status HTTP/1.1" 200 206 "-" "kube-probe/1.12"
[I]

But my kong-ingress-controller is in CrashLoopBackOf of a repeated way, this make that I have problems when I want to create new routes with new ingress.

Every 2,0s: kubectl get all -n kong                                                        el-pug: Thu Apr  4 00:38:09 2019

NAME                                           READY   STATUS             RESTARTS   AGE
pod/kong-6875478957-6b7xk                      1/1     Running            0          56m
pod/kong-ingress-controller-869bf4597c-8zrc6   2/3     CrashLoopBackOff   17         56m
pod/kong-migrations-fn6pm                      0/1     Completed          0          56m
pod/konga-85b66cffff-d2fjm                     1/1     Running            0          127m
4

Hello @bgarcial,

Were you able to solve this problem?

If this problem is reproducible, please open an issue with detailed reproducible steps so that we can replicate and fix the underlying issue. Thanks!

1 Like
5

Hi @hbagdi

My kong-ingress-controller look stable and running now but when I perform changes to the ingress resources which point to it, sometimes have a CrashLoopBackOff status and this happens with certain frequency.
The changes performed in the ingress resources which point to it, often create and remove routes, certificates and validate orders. Nothing weird.

I am not sure exactly about what are the causes of this behavior.
Let me think about and execute some tests in order to clarify this behavior and write of a detailed way the issue that you advise me in case of being necessary

6

Hi @hbagdi.

I have write an issue detailing the behavior and the environment of this situation.
It is here: Unexpected response searching a Kong Certificate: connect: connection refuse · Issue #267 · Kong/kubernetes-ingress-controller · GitHub
The kong-ingress-controller getting down continuously.

7

I was using a kong YAML manifest customized, which I didn’t add it some specific configurations with relation to the change in the URL for Kong admin API server value ( -kong-url ) which it’s recommended to be setted to http://localhost:8001 byt he author of one helper which I am using, it is: https://github.com/ollystephens/acme-kong-kube-helper

And of course I did that, I’ve changed the port to 8001, but I had to change this port value in other parts of my script, not only in the ingress-controller deployment resource, also in the ingress-controller and kong-proxy Services, and the other resource, along of the YAML manifest script.

And also I had to set the script in some parts to HTTP scheme and not HTTPS scheme, in ingress-controller deployment resource.

So, my Kong YAML customized manifest finally has stayed of this way.

I am going to close the issue that I have created at github page.