Data Plane can not authenticate against Control Plane using client certificate

Currently I’m trying to setup Kong Enterprise (trial) in Hybrid Mode. The DP is not able to establish a connection with the CP. DP and CP are running in different Kubernetes Clusters.

I get the following error in DP (replaced the real domain with example.com):

[error] 26#0: *9 [lua] data_plane.lua:232: communicate(): [clustering] 
  connection to control plane wss://cluster.kong.example.com/v1/outlet?node_id=bbd5f0b3-a601-406c-9a8a-53630929c375&node_hostname=kong-data-plane-kong-689dc7bf84-2mctr&node_version=2.5.0.0-enterprise-edition 
  broken: failed to connect: timeout (retrying after 9 seconds) 
  [cluster.kong.example.com], context: ngx.timer

For this request I don’t get any logs on the CP.

Also, I’m able to send requests using Postman to the CP (even WSS requests are possible - but unfortunately Postman does not support authorizing using client certificates for Websockets). For this request I can see the following log in the CP (for the request using Postman):

2021/08/19 01:04:51 [info] 26#0: *30223 client sent no required SSL 
  certificate while reading client request headers, client: 10.0.0.10, 
  server: kong_cluster_listener, request: "GET / HTTP/1.1", 
  host: "cluster.kong.example.com

Both Kubernetes clusters are running in GCP. I installed Kong using the Helm chart with the following configuration.

Control Plane Config:

image:
  repository: kong/kong-gateway

secretVolumes:
- kong-cluster-cert
- ca-cert

env:
  role: control_plane
  log_level: debug
  cluster_ca_cert: /etc/secrets/ca-cert/ca.crt
  cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt
  cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key
  cluster_mtls: pki
  admin_api_uri: https://admin.kong.example.com
  admin_gui_url: https://manager.kong.example.com
  database: "postgres"
  pg_user: kong
  pg_password: kong
  pg_host: "10.3.0.3"
  password:
    valueFrom:
      secretKeyRef:
        name: kong-enterprise-superuser-password
        key: password

cluster:
  enabled: true
  type: LoadBalancer
  tls:
    enabled: true
    servicePort: 443
    containerPort: 8005

proxy:
  enabled: false

enterprise:
  enabled: true
  type: ClusterIP
  rbac:
    enabled: true

manager:
  enabled: true
  type: ClusterIP
  http:
    enabled: true
    servicePort: 8002
  tls:
    enabled: true
    servicePort: 8445

admin:
  enabled: true
  type: ClusterIP
  http:
    enabled: true
    servicePort: 8001
  tls:
    enabled: true
    servicePort: 8444

# only available with enterprise license
clustertelemetry:
  enabled: true
  tls:
    enabled: true
    servicePort: 8006
    containerPort: 8006

ingressController:
  enabled: false

Data Plane Config:

image:
  repository: kong/kong-gateway

secretVolumes:
- kong-cluster-cert
- ca-cert

env:
  role: data_plane
  log_level: debug
  cluster_ca_cert: /etc/secrets/ca-cert/ca.crt
  cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt
  cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key
  cluster_mtls: pki
  lua_ssl_trusted_certificate: /etc/secrets/kong-cluster-cert/tls.crt
  cluster_server_name: cluster.kong.example.com
  cluster_control_plane: cluster.kong.example.com
  database: "off"

admin:
  enabled: false

proxy:
  enabled: true
  type: ClusterIP

ingressController:
  enabled: false

What is going wrong here? Any tips how to troubleshoot this?

1 Like

@Raman encountering the same on first stand up - any luck on the above?

FYI Resolved this issue by explicitly defining port 443 on the cluster endpoint. May be a bug in the underlying code as the wss spec should default to port 443.