When I try to use an ingress.class other than kong I am unable to use the key-auth plugin anymore. It seems like it can’t read/load the key value out of the kubernetes secret unless the ingress.class is kong (or something like that).
Following this guide verbatum except adding the annotation “kubernetes.io/ingress.class: not-kong” to each of kong’s custom resource. And when I do I am able to suddenly see the consumers + key-auth references as well as the key-auth plugins in the read-only admin API. (they won’t show without the annotation defined)
If I omit all of that and use the default ingress class of “kong” then everything works. I’m currently running KIC 0.8.1 and kong 2.0.3 deployed via the current helm chart.
Anybody have any idea what I could be missing or is it just a bug in kong itself?
Those are generally any resource type that are top-level resources when the controller creates configuration: ingresses, consumers, and global plugins all result in Kong configuration independent of links from other resources. We check credentials also even though they must be tied to a consumer–that may be a bug, but it’s only for the deprecated KongCredential CRD and not the new Secret-based credentials.
I think the ingress.class annotation thing is working for me on kong CRD. If I don’t define them then I don’t see the objects in the admin API. If I do define them I start seeing the object in the Admin API.
The problem is kong itself doesn’t see or load the kubernetes secret referenced from KongConsumer credentials and created with this command. (I’ve also tried to annotate this secret with no success)
Can you provide the consumer and secret YAML manifest (with the data portion of the secret redacted)? Do you see errors if you check the controller container logs? It should have a log like this with more detail if it fails to find one:
Interesting. The key in the request and the key in the admin API credential do indeed match, i.e. no chance it just got garbled in the secret somehow? Does the plugin configuration indeed have the key_names configured?
Are there any other plugins on the route or service that could return the 401 (e.g. ACL)?
The ability of the controller to influence request processing ends at its ability to create configuration. If the resulting configuration looks correct, something weird is going on in the Kong proxy code itself. key-authentication doesn’t have many obvious ways to fail, and as you’ve noted, doesn’t have much in the way of detailed logging (https://github.com/Kong/kong/blob/master/kong/plugins/key-auth/handler.lua only logs major errors; normal allow/deny decisions are omitted).
Are you able to dump configuration using decK, restore it on a local instance (the standard Kong Vagrant image should work well), and replicate the issue? If configuration looks correct, I usually turn to adding new debug log lines to the problem code (/usr/local/share/lua/5.1/kong/plugins/key-auth/handler.lua in this case) to try and determine the cause of the problem. That’s possible for Kubernetes deployments, but a bit cumbersome because our stock images don’t run as root (and don’t allow modifying Lua code as such). You can work around that by building a custom image that does run as root, execing into the container, editing as needed, and running kill -HUP 1 to force a reload of all Lua source.
Oddly enough I figured something out. If I limit the scope of what kong sees it starts working:
My kube-api response times are fast but there are a LOT of k8s objects for the controller to read through depending on what its looking at. Maybe there is something getting exceeded by the amount of of results KIC has to dig through?
Took me awhile to figure out that arg/env config existed for KIC. Setting it is actually desirable for us anyways.
I’m no expert so I could have missed something. But it looked like everything it should have had was there in the Admin API. (plugins, routes, services, consumers, keys, etc) I was pretty confused as to what could possibly be wrong comparing it with my other working example. If I used config.anonymous on the key-auth plugin it did honor that as expected. I just couldn’t get it to honor the consumer credential of the example above. (with both k8s secret or KongCredential methods)
The original title on this post seems like a red herring now that the namespace filter resolved it. There were 60 namespaces here if that info is helpful.