Currently I’m trying to setup Kong Enterprise (trial) in Hybrid Mode. The DP is not able to establish a connection with the CP. DP and CP are running in different Kubernetes Clusters.
I get the following error in DP (replaced the real domain with example.com):
[error] 26#0: *9 [lua] data_plane.lua:232: communicate(): [clustering]
connection to control plane wss://cluster.kong.example.com/v1/outlet?node_id=bbd5f0b3-a601-406c-9a8a-53630929c375&node_hostname=kong-data-plane-kong-689dc7bf84-2mctr&node_version=2.5.0.0-enterprise-edition
broken: failed to connect: timeout (retrying after 9 seconds)
[cluster.kong.example.com], context: ngx.timer
For this request I don’t get any logs on the CP.
Also, I’m able to send requests using Postman to the CP (even WSS requests are possible - but unfortunately Postman does not support authorizing using client certificates for Websockets). For this request I can see the following log in the CP (for the request using Postman):
2021/08/19 01:04:51 [info] 26#0: *30223 client sent no required SSL
certificate while reading client request headers, client: 10.0.0.10,
server: kong_cluster_listener, request: "GET / HTTP/1.1",
host: "cluster.kong.example.com
Both Kubernetes clusters are running in GCP. I installed Kong using the Helm chart with the following configuration.
Control Plane Config:
image:
repository: kong/kong-gateway
secretVolumes:
- kong-cluster-cert
- ca-cert
env:
role: control_plane
log_level: debug
cluster_ca_cert: /etc/secrets/ca-cert/ca.crt
cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt
cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key
cluster_mtls: pki
admin_api_uri: https://admin.kong.example.com
admin_gui_url: https://manager.kong.example.com
database: "postgres"
pg_user: kong
pg_password: kong
pg_host: "10.3.0.3"
password:
valueFrom:
secretKeyRef:
name: kong-enterprise-superuser-password
key: password
cluster:
enabled: true
type: LoadBalancer
tls:
enabled: true
servicePort: 443
containerPort: 8005
proxy:
enabled: false
enterprise:
enabled: true
type: ClusterIP
rbac:
enabled: true
manager:
enabled: true
type: ClusterIP
http:
enabled: true
servicePort: 8002
tls:
enabled: true
servicePort: 8445
admin:
enabled: true
type: ClusterIP
http:
enabled: true
servicePort: 8001
tls:
enabled: true
servicePort: 8444
# only available with enterprise license
clustertelemetry:
enabled: true
tls:
enabled: true
servicePort: 8006
containerPort: 8006
ingressController:
enabled: false
Data Plane Config:
image:
repository: kong/kong-gateway
secretVolumes:
- kong-cluster-cert
- ca-cert
env:
role: data_plane
log_level: debug
cluster_ca_cert: /etc/secrets/ca-cert/ca.crt
cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt
cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key
cluster_mtls: pki
lua_ssl_trusted_certificate: /etc/secrets/kong-cluster-cert/tls.crt
cluster_server_name: cluster.kong.example.com
cluster_control_plane: cluster.kong.example.com
database: "off"
admin:
enabled: false
proxy:
enabled: true
type: ClusterIP
ingressController:
enabled: false
What is going wrong here? Any tips how to troubleshoot this?