Intermittently receive "not secure" message and missing TLS padlock in the browser

Summary

We are using Kong 2.3 and Kong Kubernetes Ingress Controller 1.1. It is working fine, except every once in a while, we receive a missing padlock and “not secure” message in the browser. As far as we can tell, there is no rhyme nor reason to this. Seemingly randomly, every couple of weeks or so, this occurs for one of the domains we have pointing to our Kong Ingress.

The Problem (Detail)

When opening the cert (in Chrome we click on the “Not Secure” message where the padlock usually is to the left of the address bar), it says “localhost self-signed root certificate” “This certificate has not been verified by a third party” and the top info is “US California San Francisco Kong IT Department localhost”. This is strange, because our certificates normally say " Issued by: R3" and " US Let’s Encrypt R3".

At this point, we’re not sure if this issue is due to Kong, cert-manager, let’s encrypt, or something else in our cluster.

Additional Info

  1. We are using Kong Kubernetes Ingress Controller 1.1 with Helm
  2. We have installed the external-dns Helm chart, which automatically creates DNS records that point to the Kong ingress. E.g. creates A record from my-domain.com → Kong svc IP address.
  3. We have installed the cert-manager Helm chart, which automatically creates certs for any of the hosts (domains) that we have associated with our ingress. When we run kubectl get cert, all certs, including the ones that have the problem, have a valid: true property and nothing seems wrong with the cert.
  4. By simply redeploying the Helm chart of the app which is associated with the “problem domain”, the problem goes away and we get a TLS padlock again.
  5. This problem doesn’t seem to affect every domain associated with a particular app/chart. Usually it’s just one domain that has this problem on an app that could have anywhere from 2-30 domains pointing to it.
  6. The underlying apps seem to be working fine (e.g. if we connect through http unsecurely, or we shell into the container and run a curl localhost inside the container – the app works fine)

We’re really lost about where to start looking at this point. Any ideas?

Supporting images

Cert showing problem (in Google Chrome)

Cert with no problem (for comparison)

Our cert-manager chart was very out of date. Updating it solved the problem where it was not cleaning up stale ACME order CRDs.


© 2019 Kong Inc.    Terms  •  Privacy  •  FAQ