DNS resolution problem

Kubernetes
1

Hello !

We have encountered a problem, and we are seeking your knowlegde to resolve it !
The setup :

  • Kubernetes Cluster, AWS EKS, v 1.15
  • CoreDNS is deployed (by default)
  • Kong, deployed inside this cluster, connecting to PostgreSQL
  • PostgreSQL, deployed with AWS RDS

DNS resolution with CoreDNS is working fine

If we are using a short name, like postgresql, we are having this kind of error :
2020/05/18 15:47:09 [error] 22#0: *648 [kong] exporter.lua:161 prometheus: failed to reach database while processing/metrics endpoint: [cosocket] DNS resolution failed: dns server error: 3 name error. Tried: ["(short)postgresql:(na) - cache-miss","postgresql.watermarking.svc.cluster.local:5 - cache-hit/dns server error: 3 name error","postgresql.svc.cluster.local:5 - cache-hit/dns server error: 3 name error","postgresql.cluster.local:5 - cache-hit/dns server error: 3 name error","postgresql.e2e-vo-wmk.int.:5 - cache-hit/dns client error: 101 empty record received","postgresql.eu-west-1.compute.internal:5 - cache-hit/dns server error: 3 name error","postgresql:5 - cache-hit/dns server error: 3 name error","postgresql.watermarking.svc.cluster.local:33 - cache-hit/dns server error: 3 name error","postgresql.svc.cluster.local:33 - cache-hit/dns server error: 3 name error","postgresql.cluster.local:33 - cache-hit/dns server error: 3 name error","postgresql.e2e-vo-wmk.int.:33 - cache-hit/dns client error: 101 empty record received","postgresql.eu-west-1.compute.internal:33 - cache-hit/dns server error: 3 name error","postgresql:33 - cache-hit/dns server error: 3 name error","postgresql.watermarking.svc.cluster.local:1 - cache-hit/dns server error: 3 name error","postgresql.svc.cluster.local:1 - cache-hit/dns server error: 3 name error","postgresql.cluster.local:1 - cache-hit/dns server error: 3 name error","postgresql.e2e-vo-wmk.int.:1 - cache-hit/dns client error: 101 empty record received","postgresql.eu-west-1.compute.internal:1 - cache-hit/dns server error: 3 name error","postgresql:1 - cache-hit/dns server error: 3 name error"], client: 10.10.12.115, server: kong_prometheus_exporter, request: "GET /metrics HTTP/1.1", host: "10.10.11.187:9542"

If we are using a long name (FQDN), we don’t have any problem, and it’s working fine.

The thing : we want to use the short name, and since the DNS resolution is working fine, we shouldn’t have to do anything

Unless to set the DNS resolver to CoreDNS ? But it should be by default, no ?

Thank you for your time :slight_smile:

2

I’m not sure what is the question here. Can you rephrase?

3

The problem is :

For Kong there are no configuration, except database configuration (PG_HOST and other variables), and was deployed with Helm (last chart available)

  • If I use the short name (postgresql), it does not work

  • If I use the FQDN (postgresql.something.whatever), it works

  • CoreDNS resolution works fine, so the issue is not here.

So what should I do to make it work with the short name ?
Does Kong use CoreDNS in this case ?
Do I need to tune Kong settings for the DNS : https://docs.konghq.com/2.0.x/configuration/#dns-resolver-section ?

5

In that case, make sure that CoreDNS is configured as the resolver for Kong. Kong usually picks up whatever you have inside /etc/resolv.conf but it doesn’t seem to be the case in your deployment.

6

My deployment is using the default DNSPolicy, which is ClusterFirst. So searching inside the cluster first then other suffixes

I’ve checked the logs of CoreDNS to see if there was calls to him
Currently, CoreDNS is resolving alright.

Since we are using Route53 + AWS RDS, it makes the DNS resolution like this

  • postgresql.mydomain.whatever => CNAME postgresql.aws_id.rds => A IP_ADDRESS

And this line in the documentation is troubling me :

Kong will resolve hostnames as either SRV or A records (in that order, and CNAME records will be dereferenced in the process).

If I understand well, Kong will need a “A” record somewhere to work correctly, but how deep is it searching that “A” record ?

7

CNAME will be dereferenced means that Kong will follow a chain of CNAME records to reach the A record.