Dbless: Kong ingress.class != kong doesn't work with KongConsumer type: key-auth

Kubernetes
1

When I try to use an ingress.class other than kong I am unable to use the key-auth plugin anymore. It seems like it can’t read/load the key value out of the kubernetes secret unless the ingress.class is kong (or something like that).

Following this guide verbatum except adding the annotation “kubernetes.io/ingress.class: not-kong” to each of kong’s custom resource. And when I do I am able to suddenly see the consumers + key-auth references as well as the key-auth plugins in the read-only admin API. (they won’t show without the annotation defined)

With this:

If I omit all of that and use the default ingress class of “kong” then everything works. I’m currently running KIC 0.8.1 and kong 2.0.3 deployed via the current helm chart.

Anybody have any idea what I could be missing or is it just a bug in kong itself?

2

Looks to be an omission in the documentation. I’ll mark it down for us to fix.

That annotation is intentionally checked on a number of resources: if isValidIngresClass is checked, the manifest must have a matching class annotation, e.g. https://github.com/Kong/kubernetes-ingress-controller/blob/0.8.1/internal/ingress/store/store.go#L265-L277

The default kong class and empty annotation are handled by default: https://github.com/Kong/kubernetes-ingress-controller/blob/0.8.1/internal/ingress/annotations/annotations.go#L49-L61

Those are generally any resource type that are top-level resources when the controller creates configuration: ingresses, consumers, and global plugins all result in Kong configuration independent of links from other resources. We check credentials also even though they must be tied to a consumer–that may be a bug, but it’s only for the deprecated KongCredential CRD and not the new Secret-based credentials.

3

I think the ingress.class annotation thing is working for me on kong CRD. If I don’t define them then I don’t see the objects in the admin API. If I do define them I start seeing the object in the Admin API.

The problem is kong itself doesn’t see or load the kubernetes secret referenced from KongConsumer credentials and created with this command. (I’ve also tried to annotate this secret with no success)

kubectl create secret generic harry-apikey  --from-literal=kongCredType=key-auth --from-literal=key=my-sooper-secret-key

Should I try the deprecated KongCredential (plus annotation) and see if it works that way?

4

I just tried using KongCredential instead of a k8s secret and it also doesn’t work. Even though I can see it in /consumers/username/key-auth and the key is correct.

5

Can you provide the consumer and secret YAML manifest (with the data portion of the secret redacted)? Do you see errors if you check the controller container logs? It should have a log like this with more detail if it fails to find one:

error fetching credential secret 'namespace/secretname': reason"
6

Sure, here is an example with ingress.class: kong-noaa. The plugin I reference in the ingress annotations does add auth to it:

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: api-auth
  annotations:
    kubernetes.io/ingress.class: "kong-noaa"
plugin: key-auth
config:
  key_names:
    - token

So I try to create a consumer for that

---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: user2-api-token
data:
  key: ...
  kongCredType: a2V5LWF1dGg=
---
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: user2
  annotations:
    kubernetes.io/ingress.class: "kong-noaa"
username: user2
credentials:
  - "user2-api-token"

Kong is not logging any errors and seems to be able to read the key since its correct in the admin API:

curl -sS https://kong-noaa.domain/consumers/user2/key-auth | jq
{
  "next": null,
  "data": [
    {
      "created_at": 1588110704,
      "consumer": {
        "id": "e11d170c-cc4c-55b9-9f5e-f46d21416d9c"
      },
      "id": "486615f1-7c9a-5468-b9ba-2613b776ae6f",
      "tags": null,
      "key": "..."
    }
  ]
}

But if I try that with key-auth enabled on an ingress it acts like it doesn’t know anything about it (the 401 is from kong not the api behind it)

curl -sS -I -k "https://api.domain/resource?token=..."
HTTP/2 401 
date: Tue, 28 Apr 2020 22:19:24 GMT
content-type: application/json; charset=utf-8
content-length: 48
x-kong-response-latency: 1
server: kong/2.0.3
7

Interesting. The key in the request and the key in the admin API credential do indeed match, i.e. no chance it just got garbled in the secret somehow? Does the plugin configuration indeed have the key_names configured?

Are there any other plugins on the route or service that could return the 401 (e.g. ACL)?

The ability of the controller to influence request processing ends at its ability to create configuration. If the resulting configuration looks correct, something weird is going on in the Kong proxy code itself. key-authentication doesn’t have many obvious ways to fail, and as you’ve noted, doesn’t have much in the way of detailed logging (https://github.com/Kong/kong/blob/master/kong/plugins/key-auth/handler.lua only logs major errors; normal allow/deny decisions are omitted).

Are you able to dump configuration using decK, restore it on a local instance (the standard Kong Vagrant image should work well), and replicate the issue? If configuration looks correct, I usually turn to adding new debug log lines to the problem code (/usr/local/share/lua/5.1/kong/plugins/key-auth/handler.lua in this case) to try and determine the cause of the problem. That’s possible for Kubernetes deployments, but a bit cumbersome because our stock images don’t run as root (and don’t allow modifying Lua code as such). You can work around that by building a custom image that does run as root, execing into the container, editing as needed, and running kill -HUP 1 to force a reload of all Lua source.

8

Oddly enough I figured something out. If I limit the scope of what kong sees it starts working:

CONTROLLER_WATCH_NAMESPACE: mynamespace

My kube-api response times are fast but there are a LOT of k8s objects for the controller to read through depending on what its looking at. Maybe there is something getting exceeded by the amount of of results KIC has to dig through?

Took me awhile to figure out that arg/env config existed for KIC. Setting it is actually desirable for us anyways.

9

Oh, neat. @hbagdi should there be any practical limit on the number of resources the controller can successfully search?

Not sure how that would factor in if the config is visible in the admin API, but maybe there was something other than the credential that it hadn’t added.

10

I’m no expert so I could have missed something. But it looked like everything it should have had was there in the Admin API. (plugins, routes, services, consumers, keys, etc) I was pretty confused as to what could possibly be wrong comparing it with my other working example. If I used config.anonymous on the key-auth plugin it did honor that as expected. I just couldn’t get it to honor the consumer credential of the example above. (with both k8s secret or KongCredential methods)

The original title on this post seems like a red herring now that the namespace filter resolved it. There were 60 namespaces here if that info is helpful.