Konnect dataplane certificate update

Hi All,
We are trying to update the certificate at dataplane configured as part of runtime instance creation by following Renew Certificates for a Data Plane Node | Kong Docs

While updating the certificagtes we are ended up with errors.

C:\Users\Dell> echo "KONG_CLUSTER_CERT=/kong-new.crt \

KONG_CLUSTER_CERT_KEY=/private.key
kong reload exit" | docker exec -i eaf62de245a4 /bin/sh -vv
KONG_CLUSTER_CERT=/kong-new.crt
KONG_CLUSTER_CERT_KEY=/private.key
kong reload exit
Error: cluster_cert: failed loading certificate from /kong-new.crt

Run with --v (verbose) or --vv (debug) for more details

Please share any successful steps carried out in this regards.

Regards,
Hari

First I used the quickstart to create a docker based dataplane on my workstation:

docker run -d \
-e "KONG_ROLE=data_plane" \
-e "KONG_DATABASE=off" \
-e "KONG_VITALS=off" \
-e "KONG_NGINX_WORKER_PROCESSES=1" \
-e "KONG_CLUSTER_MTLS=pki" \
-e "KONG_CLUSTER_CONTROL_PLANE=..." \
-e "KONG_CLUSTER_SERVER_NAME=..." \
-e "KONG_CLUSTER_TELEMETRY_ENDPOINT=..." \
-e "KONG_CLUSTER_TELEMETRY_SERVER_NAME=..." \
-e "KONG_CLUSTER_CERT=<auto generated cert>" \
-e "KONG_CLUSTER_CERT_KEY=<auto generated key>" \
-e "KONG_LUA_SSL_TRUSTED_CERTIFICATE=system" \
-e "KONG_KONNECT_MODE=on" \
-e "KONG_CLUSTER_DP_LABELS=created-by:quickstart,type:docker-macOS" \
-p 8000:8000 \
-p 8443:8443 \
kong/kong-gateway:3.3

Then create certificates, this is using the Konnect API example from the Renew DP doc directly:

openssl req -new -x509 -nodes -newkey rsa:2048 -subj "/CN=kongdp/C=US" -keyout ./tls.key -out ./tls.crt

export CERT=$(awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' tls.crt)

Then POST the cert to your runtime group:

curl --request POST \
  --url https://us.api.konghq.com/v2/runtime-groups/{runtimeGroupId}/dp-client-certificates \
  --header 'Authorization: Bearer {patToken}' \
  --json '{"cert":"'$CERT'"}'

The response will have the dp cert id, keep this output to validate the last step:

{
  "item": {
    "id": "a2c2c071-57f4-44f2-bded-08c1842bc5f8",
    "created_at": 1687528107,
    "updated_at": 1687528107,
    "cert": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----\n",
    "metadata": {
      "subject": "CN=kongdp,C=US",
      "issuer": "CN=kongdp,C=US",
      "expiry": "1690119879",
      "key_usages": [
        "CERT_KEY_USAGE_TYPE_ENCIPHER_ONLY"
      ]
    }
  }
}

Now you need to update the container. Copy the certs into the container:

docker cp tls.crt {containerName}:/etc/kong/tls.crt

docker cp tls.key {containerName}:/etc/kong/tls.key

Then run this command:

echo "KONG_CLUSTER_CERT=/etc/kong/tls.crt \
  KONG_CLUSTER_CERT_KEY=/etc/kong/tls.key \
  kong reload exit" | docker exec -i  {containerName} /bin/sh

The output should be: Kong reloaded

From the Konnect API you can validate your runtime instance is referencing the new certificate.

Execute the request below to get the node instance record:

curl --request GET \
  --url https://us.api.konghq.com/v2/runtime-groups/{runtimeGroupId}/nodes \
  --header 'accept: application/json' \
  --header 'Authorization: Bearer {patToken}'

The output should have the same dp_cert_id

  {
    "items": [
      {
        "id": "x",
        "version": "3.3.0.0",
        "hostname": "x",
        "last_ping": 1687534536,
        "type": "kong-proxy",
        "created_at": 1687527801,
        "updated_at": 1687534536,
        "config_hash": "e333d31ef79a7003e333d31ef79a7003",
        "compatibility_status": {
          "state": "COMPATIBILITY_STATE_FULLY_COMPATIBLE"
        },
        "data_plane_cert_id": "a2c2c071-57f4-44f2-bded-08c1842bc5f8", <-- dp cert id should match id from dp cert POST response
        "labels": {
          "created-by": "quickstart",
          "type": "docker-macOS"
        }
      }
    ],
    "page": {
      "total_count": 1,
      "next_cursor": "K1gQX1APF0VFV0ZTEw19FA4UMWZWW10VDUMMVxMXExIHQUEATg=="
    }
  }