How to rotate mtls certificates for upstream service

Kubernetes
1

Hello,

I have succesfully setup Kong Ingress with mTLS for upstream services using env vars:

KONG_NGINX_PROXY_PROXY_SSL_CERTIFICATE="/path/to/client_cert.pem"
KONG_NGINX_PROXY_PROXY_SSL_CERTIFICATE_KEY="/path/to/key.pem"

as specified here: Using mTLS with Kong - v1.1.x | Kong - Open-Source API Management and Microservice Management

The certificates are mounted in a Volume and I am using a DBLess kong ingress.
The problem is, when I rotate certificates in the mounted volume configured in the environment variables, I have not found a way to successfully signal Kong to reload certificates.

What I have tried so far is:

  • Signaling SIGHUP to nginx master process: Does not work because the kong’s configuration gets lost and routes are emptied.
  • Kong reload with kubectl exec -n kong ingress-controller-pod -c proxy -- kong reload : It also does not work for the same reason, the configuration get lost and routes no longer work.

How can I do it?
Since I have already develop the infrastructure to rotate certs in the mounted volume using a sidecar container, I would appreciate if you can guide me in achieving it in in that way.

Thanks!

2

There’s no good way to handle it live in DB-less mode, since pushing config is entirely up to the controller, and there’s no designated way to manually trigger a push.

You could work around that by making a change to an Ingress that you don’t actually use, but there’d still be some lag time where the Pod was marked Ready but hadn’t received configuration yet after the worker restarts. Using a route (instead of the status endpoint) for the proxy’s readiness probe could work around that, but it’d still be imperfect, as those checks aren’t instant either.

The only safe option I can think of with the filesystem mTLS certificates is to restart the Pods entirely, so that they pick up the new certificate at start and don’t become ready/replace the old Pods until they’ve received their first config push:

kubectl rollout restart deployment YOUR_DEPLOYMENT

Enterprise provides a plugin mTLS option that allows it to update that configuration without worker restarts.