Invalid authentication credentials when following guide

davix · January 19, 2020, 12:16pm

Learning Kong for Kubernetes by following official guide: https://github.com/Kong/kubernetes-ingress-controller/blob/master/docs/guides/using-consumer-credential-resource.md

Here’s the result I got:

➜ ~ curl -i -H ‘apikey: my-sooper-secret-key’ $PROXY_IP/foo/status/200
HTTP/1.1 401 Unauthorized
Date: Sun, 19 Jan 2020 11:53:58 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Content-Length: 48
Server: kong/1.3.0

{“message”:“Invalid authentication credentials”}%

This is the earlier setting for key:

kubectl create secret generic harry-apikey  \
–from-literal=kongCredType=key-auth
–from-literal=key=my-sooper-secret-key

How should I troubleshoot it?
BTW. I’m using minikube.

hbagdi · January 20, 2020, 5:46pm

Did you update KongConsumer resource after creating the harry-apikey secret?
It seems like you missed that.

davix · January 21, 2020, 5:18am

I think I did.

Just do it again, then output is:

➜  ~ kubectl create secret generic harry-apikey2  \
  --from-literal=kongCredType=key-auth  \
  --from-literal=key=my-sooper-secret-key
secret/harry-apikey2 created

➜  ~ echo "apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: harry
username: harry
credentials:
- harry-apikey2" | kubectl apply -f -
kongconsumer.configuration.konghq.com/harry configured

➜  ~ curl -i -H 'apikey: my-sooper-secret-key' $PROXY_IP/foo/status/200
HTTP/1.1 401 Unauthorized
Date: Tue, 21 Jan 2020 05:34:39 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Content-Length: 48
Server: kong/1.3.0

{"message":"Invalid authentication credentials"}%

Is there any troubleshooting docs?

hbagdi · January 21, 2020, 7:04pm

Can you port forward 8444 of the ingress-kong container in Kong namespace and then do:

curl -k https://localhost:8444/consumers
curl -k https://localhost:8444/key-auths

We need to verify if the resources are correctly synced to Kong or not.

davix · January 22, 2020, 8:55am

/ $ curl -k https://localhost:8444/consumers
{"next":null,"data":[{"custom_id":null,"created_at":1579681662,"id":"eaec1f83-41fb-599c-8dcd-cb7750410e35","tags":null,"username":"harry"}]}/ $
/ $
/ $ curl -k https://localhost:8444/key-auths
{"next":null,"data":[]}/ $

$ curl -k https://localhost:8444/consumers/harry/key-auth
{"next":null,"data":[]}
$ k describe kongconsumer.configuration.konghq.com/harry
Name:         harry
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"configuration.konghq.com/v1","credentials":["harry-apikey"],"kind":"KongConsumer","metadata":{"annotations":{},"name":"harr...
API Version:  configuration.konghq.com/v1
Credentials:
  harry-apikey
Kind:  KongConsumer

davix · January 22, 2020, 9:25am

BTW. the traffic is http, while it’s said that port 8444 is for https. Is above check valid?

hbagdi · January 22, 2020, 5:04pm

Yes. The administration API has got nothing to do with the protocol of the traffic.

Do you have a custom Ingress class that you are using?

Can you share the logs of the Ingress Controller?
kubectl logs -f -n kong <pod-name> ingress-controller

davix · January 23, 2020, 12:12am

  Repository: git@github.com:kong/kubernetes-ingress-controller.git
  Go:         go1.13.1
-------------------------------------------------------------------------------

I0122 08:02:25.426153       1 main.go:362] Creating API client for https://10.96.0.1:443
I0122 08:02:25.433514       1 main.go:406] Running in Kubernetes Cluster version v1.17 (v1.17.0) - git (clean) commit 70132b0f130acc0bed193d9ba59dd186f0e634cf - platform
 linux/amd64
I0122 08:02:25.610130       1 main.go:148] kong version: 1.3.1
I0122 08:02:25.610249       1 main.go:157] Kong datastore: off
I0122 08:02:25.717580       1 controller.go:242] starting Ingress controller
E0122 08:02:25.717881       1 main.go:310] error running the admission controller server:open /admission-webhook/tls.crt: no such file or directory
I0122 08:02:25.730708       1 status.go:201] new leader elected: ingress-kong-68f878bbcc-k477c
I0122 08:02:25.740644       1 controller.go:135] successfully synced configuration to Kong
I0122 08:02:29.052362       1 controller.go:135] successfully synced configuration to Kong
I0122 08:02:32.385301       1 controller.go:135] successfully synced configuration to Kong
I0122 08:02:35.718711       1 controller.go:135] successfully synced configuration to Kong
I0122 08:12:25.629349       1 controller.go:135] successfully synced configuration to Kong
I0122 08:12:28.962891       1 controller.go:135] successfully synced configuration to Kong
I0122 08:12:32.296300       1 controller.go:135] successfully synced configuration to Kong
I0122 08:19:27.220494       1 controller.go:135] successfully synced configuration to Kong
I0122 08:19:30.554416       1 controller.go:135] successfully synced configuration to Kong
W0122 08:20:06.613040       1 parser.go:772] service default/httpbin does not have any active endpoints
I0122 08:20:06.648497       1 controller.go:135] successfully synced configuration to Kong
W0122 08:22:25.624203       1 parser.go:772] service default/httpbin does not have any active endpoints
I0122 08:22:25.624480       1 controller.go:135] successfully synced configuration to Kong
W0122 08:22:28.959016       1 parser.go:772] service default/httpbin does not have any active endpoints
I0122 08:22:28.959110       1 controller.go:135] successfully synced configuration to Kong
W0122 08:22:32.291690       1 parser.go:772] service default/httpbin does not have any active endpoints
I0122 08:22:32.291822       1 controller.go:135] successfully synced configuration to Kong
I0122 08:52:28.961688       1 controller.go:135] successfully synced configuration to Kong
I0122 08:52:32.295450       1 controller.go:135] successfully synced configuration to Kong
I0122 09:02:25.627401       1 controller.go:135] successfully synced configuration to Kong
E0122 09:02:25.728605       1 reports.go:107] error sending report: write udp 172.17.0.21:60175->34.195.241.46:61829: write: connection refused
I0122 09:02:28.961050       1 controller.go:135] successfully synced configuration to Kong
I0122 09:02:32.294539       1 controller.go:135] successfully synced configuration to Kong
I0122 10:52:32.302757       1 controller.go:135] successfully synced configuration to Kong
I0122 11:02:25.636006       1 controller.go:135] successfully synced configuration to Kong
E0122 11:02:25.728838       1 reports.go:107] error sending report: write udp 172.17.0.21:60175->34.195.241.46:61829: write: connection refused
I0122 11:02:28.969633       1 controller.go:135] successfully synced configuration to Kong

Then all remaining ones are
I0123 00:02:25.692154 1 controller.go:135] successfully synced configuration to Kong
I0123 00:02:29.025786 1 controller.go:135] successfully synced configuration to Kong
I0123 00:02:32.358989 1 controller.go:135] successfully synced configuration to Kong

davix · January 23, 2020, 12:23am

btw. I see the same problem on different setups (minikube or k8s on bare metal).

hbagdi · January 23, 2020, 4:32pm

I’m not sure what is going on here.
Can you share the Deployment manifest for Kong?

davix · January 25, 2020, 3:55am

I’m using the official kong-all-in-one-dbless.yaml.

Will below info be useful?

k describe deploy/ingress-kong -n kong
Name:                   ingress-kong
Namespace:              kong
CreationTimestamp:      Wed, 22 Jan 2020 22:12:41 +0800
Labels:                 app=ingress-kong
Annotations:            deployment.kubernetes.io/revision: 1
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"ingress-kong"},"name":"ingress-kong","namespace"...
Selector:               app=ingress-kong
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=ingress-kong
  Annotations:      prometheus.io/port: 9542
                    prometheus.io/scrape: true
                    traffic.sidecar.istio.io/includeInboundPorts:
  Service Account:  kong-serviceaccount
  Containers:
   proxy:
    Image:       kong:1.3
    Ports:       8000/TCP, 8443/TCP, 9542/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP
    Liveness:    http-get http://:9001/health delay=30s timeout=1s period=10s #success=1 #failure=3
    Readiness:   http-get http://:9001/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      KONG_DATABASE:                off
      KONG_NGINX_WORKER_PROCESSES:  1
      KONG_NGINX_HTTP_INCLUDE:      /kong/servers.conf
      KONG_ADMIN_ACCESS_LOG:        /dev/stdout
      KONG_ADMIN_ERROR_LOG:         /dev/stderr
      KONG_ADMIN_LISTEN:            127.0.0.1:8444 ssl
      KONG_PROXY_LISTEN:            0.0.0.0:8000, 0.0.0.0:8443 ssl http2
    Mounts:
      /kong from kong-server-blocks (rw)
   ingress-controller:
    Image:      kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller:0.6.2
    Port:       8080/TCP
    Host Port:  0/TCP
    Args:
      /kong-ingress-controller
      --kong-url=https://localhost:8444
      --admin-tls-skip-verify
      --publish-service=kong/kong-proxy
    Liveness:   http-get http://:10254/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:10254/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:        (v1:metadata.name)
      POD_NAMESPACE:   (v1:metadata.namespace)
    Mounts:           <none>
  Volumes:
   kong-server-blocks:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kong-server-blocks
    Optional:  false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   ingress-kong-5d4d5c5bd6 (1/1 replicas created)
Events:          <none>

hbagdi · January 27, 2020, 4:25pm

Please use Ingress Controller 0.7. You are using 0.6.2.

davix · January 29, 2020, 8:28am

Thank you Harry, this solves my issue!

So I was using an old version. Here’s the diff:

diff kong-all-in-one-dbless.yaml ~/Downloads/all-in-one-dbless.yaml
298a299,300
>             - grpc
>             - grpcs
522c524
<         image: kong:1.3
---
>         image: kong:1.4
536c538
<           initialDelaySeconds: 30
---
>           initialDelaySeconds: 5
556a559
>           initialDelaySeconds: 5
581c584
<         image: kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller:0.6.2
---
>         image: kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller:0.7.0
588a592
>           initialDelaySeconds: 5
602a607
>           initialDelaySeconds: 5

I didn’t suspect version issue because I thought the simple guide won’t be version specific.

Maybe we can add note on the page that >= 0.7.0 is required?

hbagdi · January 29, 2020, 3:58pm

We certainly should. PR welcome to add it!

MarsForever · April 12, 2020, 4:07pm

minikube version
minikube version: v1.6.0
commit: 11630973e5aacb8f16d706f97683bf70fffcba3b

kubectl version --short
Client Version: v1.17.0
Server Version: v1.17.0

I tried to follow the guide
https://github.com/Kong/kubernetes-ingress-controller/blob/master/docs/guides/using-consumer-credential-resource.md

kubectl create -f https://bit.ly/k4k8s
change loadbalancer to nodeport

nodePort: 30080

name: proxy-ssl
port: 443
protocol: TCP
targetPort: 8443
selector:
app: ingress-kong

type: LoadBalancer

type: NodePort

curl -i -H ‘apikey: my-sooper-secret-key’ $PROXY_IP/foo/status/200
HTTP/1.1 401 Unauthorized
Date: Sun, 12 Apr 2020 16:02:03 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Content-Length: 48
X-Kong-Response-Latency: 1
Server: kong/2.0.3

traines · April 13, 2020, 5:14pm

@ MarsForever can you create a new thread? This one is rather old, and while the issue symptoms are similar, the root cause may be different.

When you create it, please include your controller version and copies of your current consumer and credential (output from kubectl get RESOURCETYPE RESOURCENAME -o yaml).

Topic		Replies	Views
Invalid authentication credentials - basic-auth Questions	0	407	December 20, 2020
Multiple credentials for basic-auth with ingress-controller Questions	4	532	March 16, 2020
JWT Plugin: Invalid Signature	4	1210	June 27, 2020
OIDC on Kong Manager Questions	1	124	February 20, 2024
Scoping of KongCredentials Kubernetes	1	1061	April 4, 2019

Invalid authentication credentials when following guide

Related Topics