I’m trying to setup KIC in hybrid mode where some data plane nodes are outside of control plane k8s cluster. Data plane node local to the control plane connects to CONTROLLER_KONG_ADMIN_URL fine as long as it is set to Kong admin service URL (https://<svc_url>:8444). I have exposed admin API externally and secured it with auth plugin and I can connect to it over the Internet with curl and correct credentials with no problems. However, if I set CONTROLLER_KONG_ADMIN_URL and CONTROLLER_KONG_ADMIN_HEADER on data node external k8s cluster the data node fails to start with level=info msg="retry 1 to fetch metadata from kong: 403 Forbidden {\n \"message\":\"Forbidden\"\n}" error.
Yeah, I think I was (still am to a certain extent) confused by some of the deployment options. I thought of having a KIC deployment with a single control plane running in one k8s cluster and data planes running on external k8s clusters. Idea was to centralise consumers and credentials so I wouldn’t need to manage them separately for each k8s cluster. I can see now that it is not going to work and I need to have separate KIC per k8s cluster and managed shared entities in a different way.