Invalid authentication credentials when following guide

Learning Kong for Kubernetes by following official guide: https://github.com/Kong/kubernetes-ingress-controller/blob/master/docs/guides/using-consumer-credential-resource.md

Here’s the result I got:

➜ ~ curl -i -H ‘apikey: my-sooper-secret-key’ $PROXY_IP/foo/status/200
HTTP/1.1 401 Unauthorized
Date: Sun, 19 Jan 2020 11:53:58 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Content-Length: 48
Server: kong/1.3.0

{“message”:“Invalid authentication credentials”}%

This is the earlier setting for key:

kubectl create secret generic harry-apikey  \

–from-literal=kongCredType=key-auth
–from-literal=key=my-sooper-secret-key

How should I troubleshoot it?
BTW. I’m using minikube.

Did you update KongConsumer resource after creating the harry-apikey secret?
It seems like you missed that.

I think I did.

Just do it again, then output is:

➜  ~ kubectl create secret generic harry-apikey2  \
  --from-literal=kongCredType=key-auth  \
  --from-literal=key=my-sooper-secret-key
secret/harry-apikey2 created

➜  ~ echo "apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: harry
username: harry
credentials:
- harry-apikey2" | kubectl apply -f -
kongconsumer.configuration.konghq.com/harry configured

➜  ~ curl -i -H 'apikey: my-sooper-secret-key' $PROXY_IP/foo/status/200
HTTP/1.1 401 Unauthorized
Date: Tue, 21 Jan 2020 05:34:39 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Content-Length: 48
Server: kong/1.3.0

{"message":"Invalid authentication credentials"}%

Is there any troubleshooting docs?

Can you port forward 8444 of the ingress-kong container in Kong namespace and then do:

curl -k https://localhost:8444/consumers
curl -k https://localhost:8444/key-auths

We need to verify if the resources are correctly synced to Kong or not.

/ $ curl -k https://localhost:8444/consumers
{"next":null,"data":[{"custom_id":null,"created_at":1579681662,"id":"eaec1f83-41fb-599c-8dcd-cb7750410e35","tags":null,"username":"harry"}]}/ $
/ $
/ $ curl -k https://localhost:8444/key-auths
{"next":null,"data":[]}/ $

$ curl -k https://localhost:8444/consumers/harry/key-auth
{"next":null,"data":[]}
$ k describe kongconsumer.configuration.konghq.com/harry
Name:         harry
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"configuration.konghq.com/v1","credentials":["harry-apikey"],"kind":"KongConsumer","metadata":{"annotations":{},"name":"harr...
API Version:  configuration.konghq.com/v1
Credentials:
  harry-apikey
Kind:  KongConsumer

BTW. the traffic is http, while it’s said that port 8444 is for https. Is above check valid?

Yes. The administration API has got nothing to do with the protocol of the traffic.

Do you have a custom Ingress class that you are using?

Can you share the logs of the Ingress Controller?
kubectl logs -f -n kong <pod-name> ingress-controller

  Repository: git@github.com:kong/kubernetes-ingress-controller.git
  Go:         go1.13.1
-------------------------------------------------------------------------------

I0122 08:02:25.426153       1 main.go:362] Creating API client for https://10.96.0.1:443
I0122 08:02:25.433514       1 main.go:406] Running in Kubernetes Cluster version v1.17 (v1.17.0) - git (clean) commit 70132b0f130acc0bed193d9ba59dd186f0e634cf - platform
 linux/amd64
I0122 08:02:25.610130       1 main.go:148] kong version: 1.3.1
I0122 08:02:25.610249       1 main.go:157] Kong datastore: off
I0122 08:02:25.717580       1 controller.go:242] starting Ingress controller
E0122 08:02:25.717881       1 main.go:310] error running the admission controller server:open /admission-webhook/tls.crt: no such file or directory
I0122 08:02:25.730708       1 status.go:201] new leader elected: ingress-kong-68f878bbcc-k477c
I0122 08:02:25.740644       1 controller.go:135] successfully synced configuration to Kong
I0122 08:02:29.052362       1 controller.go:135] successfully synced configuration to Kong
I0122 08:02:32.385301       1 controller.go:135] successfully synced configuration to Kong
I0122 08:02:35.718711       1 controller.go:135] successfully synced configuration to Kong
I0122 08:12:25.629349       1 controller.go:135] successfully synced configuration to Kong
I0122 08:12:28.962891       1 controller.go:135] successfully synced configuration to Kong
I0122 08:12:32.296300       1 controller.go:135] successfully synced configuration to Kong
I0122 08:19:27.220494       1 controller.go:135] successfully synced configuration to Kong
I0122 08:19:30.554416       1 controller.go:135] successfully synced configuration to Kong
W0122 08:20:06.613040       1 parser.go:772] service default/httpbin does not have any active endpoints
I0122 08:20:06.648497       1 controller.go:135] successfully synced configuration to Kong
W0122 08:22:25.624203       1 parser.go:772] service default/httpbin does not have any active endpoints
I0122 08:22:25.624480       1 controller.go:135] successfully synced configuration to Kong
W0122 08:22:28.959016       1 parser.go:772] service default/httpbin does not have any active endpoints
I0122 08:22:28.959110       1 controller.go:135] successfully synced configuration to Kong
W0122 08:22:32.291690       1 parser.go:772] service default/httpbin does not have any active endpoints
I0122 08:22:32.291822       1 controller.go:135] successfully synced configuration to Kong
I0122 08:52:28.961688       1 controller.go:135] successfully synced configuration to Kong
I0122 08:52:32.295450       1 controller.go:135] successfully synced configuration to Kong
I0122 09:02:25.627401       1 controller.go:135] successfully synced configuration to Kong
E0122 09:02:25.728605       1 reports.go:107] error sending report: write udp 172.17.0.21:60175->34.195.241.46:61829: write: connection refused
I0122 09:02:28.961050       1 controller.go:135] successfully synced configuration to Kong
I0122 09:02:32.294539       1 controller.go:135] successfully synced configuration to Kong
I0122 10:52:32.302757       1 controller.go:135] successfully synced configuration to Kong
I0122 11:02:25.636006       1 controller.go:135] successfully synced configuration to Kong
E0122 11:02:25.728838       1 reports.go:107] error sending report: write udp 172.17.0.21:60175->34.195.241.46:61829: write: connection refused
I0122 11:02:28.969633       1 controller.go:135] successfully synced configuration to Kong

Then all remaining ones are
I0123 00:02:25.692154 1 controller.go:135] successfully synced configuration to Kong
I0123 00:02:29.025786 1 controller.go:135] successfully synced configuration to Kong
I0123 00:02:32.358989 1 controller.go:135] successfully synced configuration to Kong

btw. I see the same problem on different setups (minikube or k8s on bare metal).

I’m not sure what is going on here.
Can you share the Deployment manifest for Kong?

I’m using the official kong-all-in-one-dbless.yaml.

Will below info be useful?

k describe deploy/ingress-kong -n kong
Name:                   ingress-kong
Namespace:              kong
CreationTimestamp:      Wed, 22 Jan 2020 22:12:41 +0800
Labels:                 app=ingress-kong
Annotations:            deployment.kubernetes.io/revision: 1
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"ingress-kong"},"name":"ingress-kong","namespace"...
Selector:               app=ingress-kong
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=ingress-kong
  Annotations:      prometheus.io/port: 9542
                    prometheus.io/scrape: true
                    traffic.sidecar.istio.io/includeInboundPorts:
  Service Account:  kong-serviceaccount
  Containers:
   proxy:
    Image:       kong:1.3
    Ports:       8000/TCP, 8443/TCP, 9542/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP
    Liveness:    http-get http://:9001/health delay=30s timeout=1s period=10s #success=1 #failure=3
    Readiness:   http-get http://:9001/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      KONG_DATABASE:                off
      KONG_NGINX_WORKER_PROCESSES:  1
      KONG_NGINX_HTTP_INCLUDE:      /kong/servers.conf
      KONG_ADMIN_ACCESS_LOG:        /dev/stdout
      KONG_ADMIN_ERROR_LOG:         /dev/stderr
      KONG_ADMIN_LISTEN:            127.0.0.1:8444 ssl
      KONG_PROXY_LISTEN:            0.0.0.0:8000, 0.0.0.0:8443 ssl http2
    Mounts:
      /kong from kong-server-blocks (rw)
   ingress-controller:
    Image:      kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller:0.6.2
    Port:       8080/TCP
    Host Port:  0/TCP
    Args:
      /kong-ingress-controller
      --kong-url=https://localhost:8444
      --admin-tls-skip-verify
      --publish-service=kong/kong-proxy
    Liveness:   http-get http://:10254/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:10254/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:        (v1:metadata.name)
      POD_NAMESPACE:   (v1:metadata.namespace)
    Mounts:           <none>
  Volumes:
   kong-server-blocks:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kong-server-blocks
    Optional:  false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   ingress-kong-5d4d5c5bd6 (1/1 replicas created)
Events:          <none>

Please use Ingress Controller 0.7. You are using 0.6.2.

Thank you Harry, this solves my issue!

So I was using an old version. Here’s the diff:

diff kong-all-in-one-dbless.yaml ~/Downloads/all-in-one-dbless.yaml
298a299,300
>             - grpc
>             - grpcs
522c524
<         image: kong:1.3
---
>         image: kong:1.4
536c538
<           initialDelaySeconds: 30
---
>           initialDelaySeconds: 5
556a559
>           initialDelaySeconds: 5
581c584
<         image: kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller:0.6.2
---
>         image: kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller:0.7.0
588a592
>           initialDelaySeconds: 5
602a607
>           initialDelaySeconds: 5

I didn’t suspect version issue because I thought the simple guide won’t be version specific.

Maybe we can add note on the page that >= 0.7.0 is required?

We certainly should. PR welcome to add it!

minikube version
minikube version: v1.6.0
commit: 11630973e5aacb8f16d706f97683bf70fffcba3b

kubectl version --short
Client Version: v1.17.0
Server Version: v1.17.0

I tried to follow the guide

kubectl create -f https://bit.ly/k4k8s
change loadbalancer to nodeport

  • nodePort: 30080
  • name: proxy-ssl
    port: 443
    protocol: TCP
    targetPort: 8443
    selector:
    app: ingress-kong
  • type: LoadBalancer
  • type: NodePort

curl -i -H ‘apikey: my-sooper-secret-key’ $PROXY_IP/foo/status/200
HTTP/1.1 401 Unauthorized
Date: Sun, 12 Apr 2020 16:02:03 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Content-Length: 48
X-Kong-Response-Latency: 1
Server: kong/2.0.3

@ MarsForever can you create a new thread? This one is rather old, and while the issue symptoms are similar, the root cause may be different.

When you create it, please include your controller version and copies of your current consumer and credential (output from kubectl get RESOURCETYPE RESOURCENAME -o yaml).


© 2019 Kong Inc.    Terms  •  Privacy  •  FAQ