Guide for configuring ACNE plugin on k8s

Kubernetes
1

Hi,

Can you please create a guideline for installing ACNE plugin with kong ingress controller in db-less mode? The documentation on the Kong website is not so clear. I encountered many errors during the installation with the Kong helm chart.

Thanks,

2

What errors specifically? What document(s) were you following, and where did you get stuck?

https://github.com/Kong/kubernetes-ingress-controller/blob/master/docs/guides/using-kongplugin-resource.md is the general plugin guide–we don’t have one for the ACME plugin specifically, but the same concepts should apply to any plugin. https://github.com/Kong/kong-plugin-acme covers that plugin’s settings.

3

Thanks for the response.

Let me describe it in detail:

The first problem is this:
According to https://github.com/Kong/kong-plugin-acme, the plugin installation requires adding lua_ssl_trusted_certificate to kong.conf file
or
according to this document (https://docs.konghq.com/hub/kong-inc/acme/), it requires adding nginx_proxy_lua_ssl_trusted_certificate to kong.conf file.

I installed the kong ingress controller successfully with DB-less mode. I enabled admin API without tls, only Http. I tried to pass both variables that I mentioned above separately. I passed them under “env” as it is mentioned in the chart readme file. But neither of them worked. Kong gateway pod is going into CrashBackLoop status.

The second problem:

I applied the KongPlugin which points acme plugin with below configuration :

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: "acme-plugin"
config:
  account_email: "g****@gmail.com"
  tos_accepted: true
  domains:
    - "spaarmijngeld.com"
    - "*.spaarmijngeld.com"
plugin: acme

I applied it under the same namespace as kong ingress controller is. However, when i check the kong gateway pod logs, I saw that it does not accept the plugin as a valid plugin. It was giving errors like “400 Bad Request {“fields”:{“plugins”:[{“route”:“value must be null”}…”

And the last problem is that i did not exactly understand how i can map DOMAIN/.well-known/acme-challenge value to a Route in Kong. Can you give the example of ingress yaml file, please ?

Actually, a whole working guide would be super useful for everyone. I could not find any example of applying this plugin on kong running in Kubernetes. Probably because the plugin is relatively new.

Thanks for the help in advance.

4

Having reviewed this a bit more, I’d recommend using cert-manager rather than the ACME plugin. The controller can apply certificates issued by it using the tls section of the Ingress rule, and cert-manager provides CRDs that allow managing and inspecting the certificate request lifecycle in a more Kubernetes-native fashion. The ACME plugin should still be an option, but it’s a bit more cumbersome to configure.

It has to be configured as a global plugin, which is why it’s complaining about the route not being null.

The crashloop likely stems from the certificate path being wrong: if you do end up using the ACME plugin and need to correct that, what image (e.g. Alpine, CentOS) are you using?

Without fully reviewing the implementation, the docs suggest that it doesn’t really interact with routes at all, and instead runs as a catch-all for any /.well-known/acme-challenge requests. However, you’ll need some route matching those requests to avoid them getting dumped to the 404 handler: a path-only route for /.well-known/acme-challenge with no service will ensure all ACME requests are handled. Certificates are then issued when an acme entity is present for a domain.

That does present a challenge for DB-less controller-managed instances: the controller doesn’t provide a dedicated way to handle those resources, DB-less doesn’t allow manually adding them through the admin API. As of 0.9, we have added functionality for including custom resources like acme, but that’s a bit complicated to use. Given that and the advantages of the cert-manager CRD interface, I’d recommend cert-manager over the ACME plugin in Kubernetes deployments.

5

Thanks for the detailed response.

Currently I am already using cert-manager but using the ACME plugin would be a better solution for my case if it is possible.

I doubt that the crash loop is caused by the certificate path is wrong. Because I checked the certificate path by sshing to the Kubernetes node and the certificate was there. The image is the latest ubuntu version btw. @hbagdi mentioned that there is currently an issue with the plugin. Maybe he was talking about this?

Thanks for clarifying the acme-challenge. It’s sad that I cant create a path-only route for /.well-known/acme-challenge in DB-less instance. I think the other way to add this route to every backend service that I’m running, right? That would add extra work but better than being forced to use cert-manager anyway… :slight_smile:

6

What errors appear if you check kubectl logs PODNAME -c proxy? Wrong path was a guess; the actual error should make it much clearer why Kong failed to start.

Can you elaborate why using the ACME plugin is a better option for your use case? In my view it makes more sense to use it in a non-Kubernetes environment where cert-manager isn’t an option, but for Kubernetes environments cert-manager provides better-integrated solution, and should reach the same end result.

Rethinking a bit more I’m not sure if we have a means to create service-less routes via the controller, but they’re not strictly required. Any route will handle the challenge with the plugin in place; the path-only route is just a catch-all for any requests that wouldn’t otherwise match (e.g. you want a cert for example.com, but you don’t have a blanket example.com/ route to handle the challenge request, only specific example.com/foo and example.com/bar routes). There should always be some way to handle the challenge requests, it may just not be as elegant as the service-less catchall.

7

@traines is correct on the challenge handler, it doesn’t need to be service-less route, you can attach it to any service or create a dummy service. Those requests will never reach the backend and will be terminated by the plugin. This setup is needed just to let Kong reach the stage to run plugins for that path.

8

I know this is old, I faced similar problem and somewhat solved it and just want to put it right here for those who experience this issue.

first of all you need to set this environment for your kong ingress controller (as stated in the prev comment also):

env:
  lua_ssl_trusted_certificate: "/etc/ssl/certs/ca-certificates.crt"

after that all you have to do is make a dummy ingress resource for endpoint /.well-known/acme-challenge

---
apiVersion: v1
kind: Service
metadata:
  name: acme-verification
spec:
  ports:
    - protocol: TCP
      port: 65535
      targetPort: 65535

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: acme-verification
  annotations:
    kubernetes.io/ingress.class: "kong"
spec:
  rules:
  - http:
      paths:
      - path: /.well-known/acme-challenge
        pathType: Prefix
        backend:
          service:
            name: acme-verification
            port:
              number: 65535

specifying the dummy service resource probably doesnt really matter, but thats what I do.

9

So I got this working with the Ingress Rule. Currently we are stopping our cluster every night and therefore the certificate is gone when restarting the cluster again.

I tried everything to trigger the certificate creation manually but none of that worked:

  • curl https://sub.example.com -k
  • curl kong-ip/.well-known/acme-challenge/x -H “host:sub.example.com” → failure to get a peer…

Am I missing something? Is there a way to trigger that manually?