# Kong ingress readiness/liveness probes failed when use AWS Security group for kong pods **URL:** \ **Category:** Questions\ **Tags:** kubernetes\ **Created:** [February 8, 2021, 11:05pm UTC](https://discuss.konghq.com/t/kong-ingress-readiness-liveness-probes-failed-when-use-aws-security-group-for-kong-pods/7869 "2021-02-08T23:05:45Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![burakovsky](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/burakovsky/32/2541_2.png) [@burakovsky](https://discuss.konghq.com/u/burakovsky)\ **Post date:** [February 8, 2021, 11:05pm UTC](https://discuss.konghq.com/t/kong-ingress-readiness-liveness-probes-failed-when-use-aws-security-group-for-kong-pods/7869/1 "2021-02-08T23:05:45Z")
Hello Guys, We use NLB as loadbalancer for Kong service with IP targets ([service.beta.kubernetes.io/aws-load-balancer-type:](http://service.beta.kubernetes.io/aws-load-balancer-type:) nlb-ip) and we need to attach security group to pod for restricting access to our services, exposed via kong. I followed [official guide](https://aws.amazon.com/blogs/containers/introducing-security-groups-for-pods/) for attaching security group to our Kong ingress controller pods, but unfortunately ingress controller readiness/liveness probe failed in this setup > Readiness probe failed: Get [http://10.11.21.68:10254/healthz:](http://10.11.21.68:10254/healthz:) dial tcp 10.11.21.68:10254: connect: connection refused > Liveness probe failed: Get [http://10.11.21.68:10254/healthz:](http://10.11.21.68:10254/healthz:) dial tcp 10.11.21.68:10254: connect: connection refused Rediness/liveness probes for proxy container passed. There is no any error logs, but when I run **netstat -ntlp** from ingress controller container, I don’t see 10254 port allocated: ``` tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN - tcp 0 0 127.0.0.1:8444 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:8100 0.0.0.0:* LISTEN - ``` AWS VPC CNI documentations doesn’t have any related recommendations (just add DISABLE\_TCP\_EARLY\_DEMUX=true variable), but it fixed only probes for proxy container. We use Helm chart v1.14.2 for Kong installation. ---
**Author:** ![burakovsky](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/burakovsky/32/2541_2.png) [@burakovsky](https://discuss.konghq.com/u/burakovsky)\ **Post date:** [February 15, 2021, 1:06pm UTC](https://discuss.konghq.com/t/kong-ingress-readiness-liveness-probes-failed-when-use-aws-security-group-for-kong-pods/7869/2 "2021-02-15T13:06:07Z")
I found that kong ingress-controller pod can’t connect to kubernetes api service ([https://172.20.0.1:443](https://172.20.0.1:443)) after adding security group directly to the pod. Probably that’s why probes failed. Don’t know why it can’t connect to kubernetes api service, still investigating ---
**Author:** ![burakovsky](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/burakovsky/32/2541_2.png) [@burakovsky](https://discuss.konghq.com/u/burakovsky)\ **Post date:** [February 15, 2021, 2:48pm UTC](https://discuss.konghq.com/t/kong-ingress-readiness-liveness-probes-failed-when-use-aws-security-group-for-kong-pods/7869/3 "2021-02-15T14:48:07Z")
After fixing api connection issue, currently I bumped into the next one: > level=info msg=“retry 4 to fetch metadata from kong: making HTTP request: Get "[https://localhost:8444/\](https://localhost:8444/%5C)”: context deadline exceeded" The healthy pods (without attached security group) has some similar logs, but not exactly the same: > level=info msg=“retry 2 to fetch metadata from kong: making HTTP request: Get "[https://localhost:8444/\](https://localhost:8444/%5C)”: dial tcp 127.0.0.1:8444: connect: connection refused" Both kong installation (with and without attached SG) has > admin: > enabled: false I also tried to use http instead of https for admin configuration, but it didn’t help: > level=info msg=“retry 4 to fetch metadata from kong: making HTTP request: Get "[http://localhost:8001/\](http://localhost:8001/%5C)”: context deadline exceeded" ---
**Author:** ![burakovsky](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/burakovsky/32/2541_2.png) [@burakovsky](https://discuss.konghq.com/u/burakovsky)\ **Post date:** [February 15, 2021, 4:07pm UTC](https://discuss.konghq.com/t/kong-ingress-readiness-liveness-probes-failed-when-use-aws-security-group-for-kong-pods/7869/4 "2021-02-15T16:07:12Z")
I found the issue. The problem is that pod can’t connect to CoreDNS service if I use NodeLocal DNSCache. After deleting NodeLocal DNSCache everything works correct. Looks like it’s issue with security group per pod realization. So, there is no any issue with kong, but I leave it here in case of anyone else will have the same problem ---
**Author:** ![arkrishnan](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/arkrishnan/32/2892_2.png) [@arkrishnan](https://discuss.konghq.com/u/arkrishnan)\ **Post date:** [July 22, 2021, 9:24am UTC](https://discuss.konghq.com/t/kong-ingress-readiness-liveness-probes-failed-when-use-aws-security-group-for-kong-pods/7869/5 "2021-07-22T09:24:26Z")
Hi @burakovsky, i am facing the same issue. I am using AWS CNI, and trying to get kong create a NLB. But I am getting the same below, error. ```auto Warning Unhealthy 64s (x6 over 114s) kubelet Liveness probe failed: Get "http://10.0.7.233:10254/healthz": dial tcp 10.0.7.233:10254: connect: connection refused Normal Killing 64s (x2 over 94s) kubelet Container ingress-controller failed liveness probe, will be restarted You mentioned adding security group to the pod, can you guide me through this please?? ``` ---
**Author:** ![arkrishnan](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/arkrishnan/32/2892_2.png) [@arkrishnan](https://discuss.konghq.com/u/arkrishnan)\ **Post date:** [July 22, 2021, 9:55am UTC](https://discuss.konghq.com/t/kong-ingress-readiness-liveness-probes-failed-when-use-aws-security-group-for-kong-pods/7869/6 "2021-07-22T09:55:37Z")
Also,can you tell me what was the fix for the api issue?? ---
**Author:** ![burakovsky](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/burakovsky/32/2541_2.png) [@burakovsky](https://discuss.konghq.com/u/burakovsky)\ **Post date:** [July 29, 2021, 11:35am UTC](https://discuss.konghq.com/t/kong-ingress-readiness-liveness-probes-failed-when-use-aws-security-group-for-kong-pods/7869/7 "2021-07-29T11:35:23Z")
Do you attach security group directly to kong pods or not. It should work correctly with nodelocal dnscache if you don’t attach any security group to long pods ---
**Author:** ![burakovsky](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/burakovsky/32/2541_2.png) [@burakovsky](https://discuss.konghq.com/u/burakovsky)\ **Post date:** [July 29, 2021, 12:05pm UTC](https://discuss.konghq.com/t/kong-ingress-readiness-liveness-probes-failed-when-use-aws-security-group-for-kong-pods/7869/8 "2021-07-29T12:05:32Z")
If you use NodeLocal DNSCache and attach AWS security group directly to Kong pods, you need to configure dnsPolicy and dnsConfig as described [here](https://github.com/aws/amazon-vpc-cni-k8s/issues/1384#issuecomment-856932376).