Newbie Question: ingress not properly added to kong-ingress

Kubernetes
1

Greetings,

Simple question, Why is my API not added to Kong? (NEW KUBERNETES CLUSTER FROM GCE)

  1. I have deployed the all-in-one kong-ingress controller. Everything is running fine.
  2. I have made a deployment + service for a simple API. It is also Running fine.
  3. I create an ingress pointing to my service and using Kong-ingress with a kubernetes.io/ingress.class: nginx
  4. I describe the ingress created for my API and see:
    Normal CREATE 3s (x2 over 2h) kong-ingress-controller Ingress test/test
  5. I go to URL/PATH specified in the ingress:
    The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
  6. I check postgresql’s database and SELECT * FROM apis and get 0 rows

What am I missing?

2

Could you share the Ingress spec you’re using?

3 
spec:
  rules:
  - host: dummy.kong.example
    http:
      paths:
      - backend:
          serviceName: test
          servicePort: 80
        path: /test
status:
  loadBalancer: {}
4

Do you actually have a service named test in the same namespace as of this Ingress rule?

Do you see any errors in the logs of the Ingress controller?

5

I have added the --ingress-class=kong in the kong-ingress-controller deployment. Still nothing

image

image

image

If I try to access it directly it works (kubectl port-forward)

Basicly it’s like the ingress “adds it” but does not really add it.

I am on Kubernetes Version 1.11 does it have anything to do with it?

6

Ahhh by looking at the ingress-controller pods logs I see the following errors every 15 seconds:

E1221 14:56:38.134348       6 leaderelection.go:258] Failed to update lock: configmaps "ingress-controller-leader-kong" is forbidden: User "system:serviceaccount:kong:kong-serviceaccount" cannot update configmaps in the namespace "kong"

But by looking at the role it should be able to update / get it:

image
image.png752×253 23.7 KB

image
image.png865×223 20.5 KB

7

Okay I fixed my previous problem (since I changed the ingress class name I had to add CM edit to -kong instead of -nginx)

But my main problem remains

I1221 15:09:47.669794       6 leaderelection.go:184] successfully acquired lease kong/ingress-controller-leader-kong
I1221 15:09:47.669844       6 status.go:217] new leader elected: kong-ingress-controller-bc69fb87b-926qg
I1221 15:09:47.670071       6 controller.go:128] syncing Ingress configuration...
I1221 15:09:47.705856       6 kong.go:241] creating Kong Target 10.12.1.21:5000 for upstream 0xc0002120b0
I1221 15:09:47.874494       6 kong.go:257] deleting Kong Target 0xc0003f0020 from upstream 0xc0003f0030
I1221 15:09:47.912865       6 kong.go:113] syncing global plugins
W1221 15:09:47.977453       6 kong.go:335] there is no custom Ingress configuration for rule test/test
W1221 15:09:47.992414       6 kong.go:749] there is no custom Ingress configuration for rule test/test
I1221 15:09:47.994292       6 kong.go:775] creating Kong Route for host dummy.kong.example, path /test and service c42b9513-5021-4d71-ac46-2307a95ae45d
I1221 15:09:48.008707       6 kong.go:914] deleting Kong Route 925a364f-de03-4843-a818-84be5c690add
I1221 15:09:51.003576       6 controller.go:128] syncing Ingress configuration...
I1221 15:09:51.117619       6 kong.go:113] syncing global plugins
W1221 15:09:51.120874       6 kong.go:335] there is no custom Ingress configuration for rule test/test
W1221 15:09:51.136374       6 kong.go:749] there is no custom Ingress configuration for rule test/test
I1221 15:09:51.137776       6 kong.go:803] updating Kong Route for host dummy.kong.example, path /test and service 0xc0003f0f90

But I still get a 404 when I try to access my ingress path:

image
image.png795×132 8.8 KB

8

I think my API is the problem but… when I check the apis using the kong-api api it returns 0 apis… is it normal?

curl -XGET http://127.0.0.1:8001/apis
{"total":0,"data":[]}
9

Ingress controller doesn’t use APIs in Kong. It uses Routes and services in Kong so that is indeed normal.

10

Have you set up a hosts entry in your OS to resolve dummy.kong.example to the correct network location?

11

I was gonna open a new post, but here I see the error I’m having, so I’ll try luck here.

@DevopsOP how did you fix the configmaps error? I see you need to change something because of a change of yours, but that error appears me every time I try to set-up kong with the default helm chart (only changing admin to NOT use TLS, and the readiness and liveness probes to use HTTP instead):

imatge
imatge.png1119×162 31.6 KB

It happened me a lot of times… and it’s always related to the ingress controller. I even tried to create my custom ingress controller, but results in the same error.

Can anybody please help me? :___)

12

Ok apparently the problem I have is here: https://github.com/helm/charts/blob/master/stable/kong/templates/controller-rbac-role.yaml#L31

For some reason, my configmap name is kong-ingress-controller-leader-nginx-nginx instead of kong-ingress-controller-leader-nginx (note the extra -nginx).

But, as said, I didn’t change anything that may affect this (or at least, not that I’m aware of).

I’ve been taking a look to the ingress code, and I see this https://github.com/Kong/kubernetes-ingress-controller/blob/master/internal/ingress/status/status.go#L193 which makes me think that the -nginx appending comes from there. But not really sure…

13

Manually editing the role created by the helm charts and adding kong-ingress-controller-leader-nginx-nginx the the configmaps list “fixes the issue”.

For me this is clearly a bug somewhere in the kong ingress controller definition, which is wrongly creating a configmap named kong-ingress-controller-leader-nginx-nginx instead of ingress-controller-leader-nginx.

14

Could you test out the following?

15

Thank you! That fixes the RBAC issue.

16

Glad it’s fixed the issue!

17

I am still having issues with kong ingress controller.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: kong
  creationTimestamp: 2018-12-21T13:43:23Z
  generation: 3
  name: test
  namespace: test
spec:
  rules:
  - host: kong.dev.xxxxxx.io
    http:
      paths:
      - backend:
          serviceName: test
          servicePort: 80
        path: /test
status:
  loadBalancer:
    ingress:
    - ip: <Load Balancer of Kong>

I can safely go to http://kong.dev.xxxxxx.io/ which says: {"message":"no route and no API found with those values"} and that is normal… but when I add /test (which should be the path to my api) I get a error 404 not found.

kong.dev.xxxxxx.io is correctly bound to kong’s load balancer publicly

Result from :8001/upstream

{
    "total": 1,
    "data": [
        {
            "healthchecks": {
                "active": {
                    "unhealthy": {
                        "http_statuses": [
                            429,
                            404,
                            500,
                            501,
                            502,
                            503,
                            504,
                            505
                        ],
                        "tcp_failures": 0,
                        "timeouts": 0,
                        "http_failures": 0,
                        "interval": 0
                    },
                    "http_path": "/",
                    "healthy": {
                        "http_statuses": [
                            200,
                            302
                        ],
                        "interval": 0,
                        "successes": 0
                    },
                    "timeout": 1,
                    "concurrency": 10
                },
                "passive": {
                    "unhealthy": {
                        "http_failures": 0,
                        "http_statuses": [
                            429,
                            500,
                            503
                        ],
                        "tcp_failures": 0,
                        "timeouts": 0
                    },
                    "healthy": {
                        "successes": 0,
                        "http_statuses": [
                            200,
                            201,
                            202,
                            203,
                            204,
                            205,
                            206,
                            207,
                            208,
                            226,
                            300,
                            301,
                            302,
                            303,
                            304,
                            305,
                            306,
                            307,
                            308
                        ]
                    }
                }
            },
            "created_at": 1545406120001,
            "hash_on": "none",
            "id": "d72c45f1-6dfb-4eff-9efb-30816b87285a",
            "hash_on_cookie_path": "/",
            "name": "test.test.80",
            "hash_fallback": "none",
            "slots": 10000
        }
    ]
}
18

Could you share the output of :8001/routes, :8001/services and logs of the Ingress controller?

19

/routes

{
    "next": null,
    "data": [
        {
            "created_at": 1546545758,
            "strip_path": false,
            "hosts": [
                "kong.dev.xxxxxx.io"
            ],
            "preserve_host": false,
            "regex_priority": 0,
            "updated_at": 1546545762,
            "paths": [
                "/test"
            ],
            "service": {
                "id": "1a84488d-0e67-4ef7-b77a-82eed9cc88ea"
            },
            "methods": null,
            "protocols": [
                "http"
            ],
            "id": "cec78e49-6a3b-42b6-b1d2-8c5199e78de7"
        }
    ]
}

/services

{
    "next": null,
    "data": [
        {
            "host": "test.test.80",
            "created_at": 1545406120,
            "connect_timeout": 60000,
            "id": "1a84488d-0e67-4ef7-b77a-82eed9cc88ea",
            "protocol": "http",
            "name": "test.test.80",
            "read_timeout": 60000,
            "port": 80,
            "path": "/",
            "updated_at": 1545406120,
            "retries": 5,
            "write_timeout": 60000
        }
    ]
}

ingress logs

I0103 20:33:52.226852       6 kong.go:113] syncing global plugins
W0103 20:33:52.229214       6 kong.go:335] there is no custom Ingress configuration for rule test/test
W0103 20:33:52.236795       6 kong.go:749] there is no custom Ingress configuration for rule test/test
I0103 20:43:45.450529       6 controller.go:128] syncing Ingress configuration...
I0103 20:43:45.565567       6 kong.go:113] syncing global plugins
W0103 20:43:45.568166       6 kong.go:335] there is no custom Ingress configuration for rule test/test
W0103 20:43:45.576482       6 kong.go:749] there is no custom Ingress configuration for rule test/test
I0103 20:43:48.784074       6 controller.go:128] syncing Ingress configuration...
I0103 20:43:48.897867       6 kong.go:113] syncing global plugins
W0103 20:43:48.901450       6 kong.go:335] there is no custom Ingress configuration for rule test/test
W0103 20:43:48.913334       6 kong.go:749] there is no custom Ingress configuration for rule test/test
I0103 20:43:52.117360       6 controller.go:128] syncing Ingress configuration...
I0103 20:43:52.229147       6 kong.go:113] syncing global plugins
W0103 20:43:52.231984       6 kong.go:335] there is no custom Ingress configuration for rule test/test
W0103 20:43:52.241787       6 kong.go:749] there is no custom Ingress configuration for rule test/test
I0103 20:46:07.223866       6 controller.go:128] syncing Ingress configuration...
I0103 20:46:07.333225       6 kong.go:113] syncing global plugins
W0103 20:46:07.335222       6 kong.go:335] there is no custom Ingress configuration for rule test/test
W0103 20:46:07.341480       6 kong.go:749] there is no custom Ingress configuration for rule test/test
I0103 20:46:53.825375       6 controller.go:128] syncing Ingress configuration...
I0103 20:46:53.934197       6 kong.go:113] syncing global plugins
W0103 20:46:53.936277       6 kong.go:335] there is no custom Ingress configuration for rule test/test
W0103 20:46:53.942990       6 kong.go:749] there is no custom Ingress configuration for rule test/test
20

Thanks for sharing the configs. It confirms that the Ingress controller is working correctly and populating configuration in Kong correctly.

Is it possible that the Load balancer is not forwarding the correcct host header to Kong, meaning that Kong doesn’t actually see kong.dev.xxxxxx.io header and hence can’t route the request?

A quick way to verify this will be to remove host from the Ingress spec and test if path only based routing works or not.