Azure Kubernetes Cluster with multiple kong ingress and AKIC is adding services and routes from other namespaces

Questions
1

Running in Azure, I have one kubernetes cluster with several namespaces. The cluster uses the Azure Kubernetes Ingress Controller (AKIC) - only allowed one per cluster

Each namespace has an instance of kong (database + ingress).

When I use the kubernetes ingress to define ingress to a service it is registered to all namespaces.

Each namespace uses it’s own name for the IngressClass.

If I delete the routes and services, they immediately get added back somewhere, I see the API calls for the DELETE processed in the kong ingress log, but no POST to add them back:

10.89.26.108 - - [06/Feb/2023:16:43:51 +0000] "DELETE /routes/a1f2fa03-0a94-4c29-9f97-88106537e24f HTTP/1.1" 204 0 "-" "-"

Azure Ingress definitions for namespace:

Ingress 1
Ingress 11330×316 34.5 KB

Ingress 2
Ingress 21318×319 34 KB

ingress definitions

dbetter-us-la:pavement-express-kubernetes dbetterton$ kubectl get ingress ams-web -n ams-fb-kam-871-nonprod
NAME      CLASS                         HOSTS                                ADDRESS   PORTS   AGE
ams-web   kong-ams-fb-kam-871-nonprod   fb-kam-871.nonprod.agileassets.com             80      21h
dbetter-us-la:pavement-express-kubernetes dbetterton$ kubectl get ingress ams-web -n ams-q-branch-nonprod
NAME      CLASS   HOSTS                              ADDRESS   PORTS   AGE
ams-web   kong    q-branch.nonprod.agileassets.com             80      14d
2

Services definition - note both ams-q-branch-nonprod and ams-fb-kam-871-nonprod, but only for services with the “managed-by-ingress-controller” tag, the other services, which self register using API calls are correct

{
    "data" : [
        {
            "tags" : [
                "managed-by-ingress-controller"
            ],
            "ca_certificates" : null,
            "connect_timeout" : 60000,
            "path" : "/",
            "read_timeout" : 60000,
            "protocol" : "http",
            "created_at" : 1675624432,
            "updated_at" : 1675624432,
            "host" : "ams-web.ams-q-branch-nonprod.8080.svc",
            "name" : "ams-q-branch-nonprod.ams-web.pnum-8080",
            "tls_verify" : null,
            "retries" : 5,
            "tls_verify_depth" : null,
            "client_certificate" : null,
            "port" : 80,
            "enabled" : true,
            "write_timeout" : 60000,
            "id" : "18358fec-fce0-4e88-aa20-89cc4f689b1a"
        },
        {
            "tags" : [
                "managed-by-ingress-controller"
            ],
            "ca_certificates" : null,
            "connect_timeout" : 60000,
            "path" : "/",
            "read_timeout" : 60000,
            "protocol" : "http",
            "created_at" : 1675624432,
            "updated_at" : 1675624432,
            "host" : "jasperserver.ams-fb-kam-815-nonprod.3001.svc",
            "name" : "ams-fb-kam-815-nonprod.jasperserver.pnum-3001",
            "tls_verify" : null,
            "retries" : 5,
            "tls_verify_depth" : null,
            "client_certificate" : null,
            "port" : 80,
            "enabled" : true,
            "write_timeout" : 60000,
            "id" : "1a9a5e56-93bc-4d9c-83b6-b286575c1956"
        },
        {
            "tags" : null,
            "ca_certificates" : null,
            "connect_timeout" : 60000,
            "path" : null,
            "read_timeout" : 60000,
            "protocol" : "http",
            "created_at" : 1675624519,
            "updated_at" : 1675624519,
            "host" : "10.89.24.153",
            "name" : "AmsAnalysis",
            "tls_verify" : null,
            "retries" : 5,
            "tls_verify_depth" : null,
            "client_certificate" : null,
            "port" : 8280,
            "enabled" : true,
            "write_timeout" : 60000,
            "id" : "2609fb1b-1924-4cc5-a00b-10e16fb534e3"
        },
        {
            "tags" : [
                "managed-by-ingress-controller"
            ],
            "ca_certificates" : null,
            "connect_timeout" : 60000,
            "path" : "/",
            "read_timeout" : 60000,
            "protocol" : "http",
            "created_at" : 1675624432,
            "updated_at" : 1675624432,
            "host" : "ams-ui.ams-q-branch-nonprod.80.svc",
            "name" : "ams-q-branch-nonprod.ams-ui.pnum-80",
            "tls_verify" : null,
            "retries" : 5,
            "tls_verify_depth" : null,
            "client_certificate" : null,
            "port" : 80,
            "enabled" : true,
            "write_timeout" : 60000,
            "id" : "5a8e4673-18eb-479b-937e-9da6371284d3"
        },
        {
            "tags" : [
                "managed-by-ingress-controller"
            ],
            "ca_certificates" : null,
            "connect_timeout" : 60000,
            "path" : "/",
            "read_timeout" : 60000,
            "protocol" : "http",
            "created_at" : 1675624432,
            "updated_at" : 1675624432,
            "host" : "jasperserver.ams-q-branch-nonprod.3001.svc",
            "name" : "ams-q-branch-nonprod.jasperserver.pnum-3001",
            "tls_verify" : null,
            "retries" : 5,
            "tls_verify_depth" : null,
            "client_certificate" : null,
            "port" : 80,
            "enabled" : true,
            "write_timeout" : 60000,
            "id" : "6000471b-90b1-44ae-99a8-510746e1e3cb"
        },
        {
            "tags" : [
                "managed-by-ingress-controller"
            ],
            "ca_certificates" : null,
            "connect_timeout" : 60000,
            "path" : "/",
            "read_timeout" : 60000,
            "protocol" : "http",
            "created_at" : 1675624432,
            "updated_at" : 1675624432,
            "host" : "ams-web.ams-fb-kam-815-nonprod.8080.svc",
            "name" : "ams-fb-kam-815-nonprod.ams-web.pnum-8080",
            "tls_verify" : null,
            "retries" : 5,
            "tls_verify_depth" : null,
            "client_certificate" : null,
            "port" : 80,
            "enabled" : true,
            "write_timeout" : 60000,
            "id" : "993e5cf6-5932-4b8e-b09c-dca1c98e4c34"
        },
        {
            "tags" : [
                "managed-by-ingress-controller"
            ],
            "ca_certificates" : null,
            "connect_timeout" : 60000,
            "path" : "/",
            "read_timeout" : 60000,
            "protocol" : "http",
            "created_at" : 1675624432,
            "updated_at" : 1675624432,
            "host" : "ams-ui.ams-fb-kam-815-nonprod.80.svc",
            "name" : "ams-fb-kam-815-nonprod.ams-ui.pnum-80",
            "tls_verify" : null,
            "retries" : 5,
            "tls_verify_depth" : null,
            "client_certificate" : null,
            "port" : 80,
            "enabled" : true,
            "write_timeout" : 60000,
            "id" : "d0400b24-f16b-4606-9b37-08ea7736691a"
        },
        {
            "tags" : null,
            "ca_certificates" : null,
            "connect_timeout" : 60000,
            "path" : null,
            "read_timeout" : 60000,
            "protocol" : "http",
            "created_at" : 1675624651,
            "updated_at" : 1675624651,
            "host" : "10.89.26.13",
            "name" : "AmsAnalysisData",
            "tls_verify" : null,
            "retries" : 5,
            "tls_verify_depth" : null,
            "client_certificate" : null,
            "port" : 8087,
            "enabled" : true,
            "write_timeout" : 60000,
            "id" : "ea3f7729-470f-4f05-833b-cf50f9cf8ce7"
        },
        {
            "tags" : null,
            "ca_certificates" : null,
            "connect_timeout" : 60000,
            "path" : null,
            "read_timeout" : 60000,
            "protocol" : "http",
            "created_at" : 1675624615,
            "updated_at" : 1675624615,
            "host" : "10.89.24.192",
            "name" : "AmsUiMetaData",
            "tls_verify" : null,
            "retries" : 5,
            "tls_verify_depth" : null,
            "client_certificate" : null,
            "port" : 8089,
            "enabled" : true,
            "write_timeout" : 60000,
            "id" : "f6877b8d-d620-43f7-847c-1b982a37e3df"
        }
    ],
    "next" : null
}
3

Is this the same report as Azure Kubernetes Ingress Controller enablement is adding routes from other namespaces · Issue #3504 · Kong/kubernetes-ingress-controller · GitHub ? Different person it looks like, but maybe similar issue described.

I’m unclear on whether this is asking about our controller (my comment on the issue describes our various options for namespace and non-namespace config segmentation) or the Azure controller? We don’t know much about the Azure controller.

Routes not getting re-added after you delete them is normal. KIC operates on the assumption that it is the only actor making configuration changes, and does not attempt to reconcile configuration if its expected configuration does not change. Modifying an Ingress will change the expected configuration and trigger an update, whereas modifying a Kong route directly will not change the expected configuration, and KIC will only reconcile configuration after some other Kubernetes resource changes and triggers an update.

The reverse sync CLI argument overrides this and always updates Kong regardless of changes, but isn’t recommended because it generates additional load on Kong.