SOLVED - Custom Plugin not Working With Ingress Controller

Questions
1

update

My issue ended up being that I wasn’t actually installing the lua rock on the kong image used for the admin api within the ingress controller deployment.
Once the plugin and lua path environment variables were set, it all worked!

/////////////////////////////////////////////////////////////

I am using the kong kubernetes ingress controller in gke, and am having trouble getting a custom plugin to be recognized as installed, and then configured on a service/route/etc.

I am including the plugin in my kong.conf, and have shelled into the container to verify that it is correctly configured.
I have used the exact same docker image of kong, running locally via docker-compose, and successfully applied the same custom plugin on routes to other local docker containers.

Here’s my kong.conf:

ssl = on 
ssl_cipher_suite = modern       
plugins = bundled, kong-plugin-auth-headers
ssl_cert = /etc/kong/certs/cert.crt
ssl_cert_key = /etc/kong/certs/cert.key
server_tokens = off              
latency_tokens = off            
real_ip_header = X-Forwarded-For

However, after configuring via manifest, postgres does not have an association created in the plugins table, and the ingress controller reports the following:

requeuing development/nginx-echo-headers-ingress, err creating a Kong plugin {&lt;nil&gt; &lt;nil&gt; 0xc000613a40 [ 1d59c9d9-7513-420d-b367-3b0f67185c7e nil [ DELETE, GET, OPTIONS, POST, PUT ] [ <redacted-hostname> ] [ /echo ] nil nil nil true true 0 dd7f0460-0541-48ae-81c8-4078b7d1db55 ] &lt;nil&gt; &lt;nil&gt; map[name:kong-plugin-auth-headers] &lt;nil&gt; &lt;nil&gt;} in service 0xc000213120: 400 Bad Request {"message":"schema violation (name: plugin 'kong-plugin-auth-headers' not enabled; add it to the 'plugins' configuration property)","name":"schema violation","fields":{"name":"plugin 'kong-plugin-auth-headers' not enabled; add it to the 'plugins' configuration property"},"code":2}

Here is the manifest used to deploy the ingress:

   kind: Ingress
   metadata:
     name: nginx-echo-headers-ingress
     namespace: development
     annotations:
       nginx.ingress.kubernetes.io/rewrite-target: /
       kubernetes.io/ingress.class: "kong"
       configuration.konghq.com: nginx-echo-headers-kongingress
       plugins.konghq.com: add-ratelimiting-to-route
       plugins.konghq.com: echo-auth-headers
   spec:
     tls:
     - hosts:
       - "redactedhost"
       - "redactedhost"
       - "redactedhost"
       - "redactedhost"
       - "redactedhost"
       secretName: "redacted"
     rules:  
       - host: "redactedhost"
         http:
           paths:
             - path: "/echo"
               backend:
                 serviceName: nginx-echo-headers-svc
                 servicePort: https
       - host: "redactedhost"
         http:
           paths:
             - path: "/echo"
               backend:
                 serviceName: nginx-echo-headers-svc
                 servicePort: https
       - host: "redactedhost"
         http:
           paths:
             - path: "/echo"
               backend:
                 serviceName: nginx-echo-headers-svc
                 servicePort: https
       - host: "redactedhost"
         http:
           paths:
             - path: "/echo"
               backend:
                 serviceName: nginx-echo-headers-svc
                 servicePort: https
       - host: "This text will be hiddenredactedhost"
         http:
           paths:
             - path: "/echo"
               backend:
                 serviceName: nginx-echo-headers-svc
                 servicePort: https
   ---
   apiVersion: configuration.konghq.com/v1
   kind: KongIngress
   metadata:
     name: nginx-echo-headers-kongingress
     namespace: development
   proxy:
     path: "/echo"
   route:
     methods:
     - POST
     - PUT
     - GET
     - OPTIONS
     - DELETE
     protocols:
     - https
     - http
     strip_path: true
     preserve_host: true
   ---
   apiVersion: configuration.konghq.com/v1
   kind: KongPlugin
   metadata:
     name: echo-auth-headers
     namespace: development
   plugin: kong-plugin-auth-headers
   config:
     name: kong-plugin-auth-headers
   ---
   apiVersion: configuration.konghq.com/v1
   kind: KongPlugin
   metadata:
     name: add-ratelimiting-to-route
     namespace: development
   plugin: rate-limiting
   config:
     hour: 10000
     limit_by: ip
     second: 100

Is there any other information I could offer that would be helpful in debugging?
In the meantime, I’ll be looking through and adding logging to the ingress controller code for clues as to what might be causing the issue.

Any help would be very appreciated!
Thanks,

  • Dylan