Kong Ingress Controller and Istio Service Mesh with STRICT mtls

Kubernetes
1

We have deployed Kong in Kubernetes cluster with Istio (Envoy sidecar) by following the steps documented here and from this post.

We have created an Ingress and upstream service in the same namespace as Kong proxy.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: keyless-ingress
  annotations:
    kubernetes.io/ingress.class: kong
    konghq.com/strip-path: "true"
    konghq.com/preserve-host: "false"
    #konghq.com/plugins: helloworld-plugin
spec:
  rules:
   - http: 
       paths:
         - path: /v1/keyless
           pathType: Prefix
           backend:
             service:
               name: helloworld
               port:
                 number: 5000

In the upstream service we have added below annotation -

ingress.kubernetes.io/service-upstream: “true”

Also we have enabled STRICT mtls for this namespace.

kubectl apply -n kong -f - <<EOF
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
spec:
  mtls:
    mode: STRICT
EOF

When I call this API I am getting SSL errors. Below is the error log coming in proxy -

2021/05/13 11:59:19 [info] 22#0: *5333 client closed connection while SSL handshaking, client: 127.0.0.1, server: 0.0.0.0:8443

Below are the curl verbose logs -

* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: ca.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS alert, unknown CA (560):
* error:1401E418:SSL routines:CONNECT_CR_FINISHED:tlsv1 alert unknown ca
* Closing connection 0
curl: (35) error:1401E418:SSL routines:CONNECT_CR_FINISHED:tlsv1 alert unknown ca

When I change the mtls mode to PERMISSIVE then I am able to call the API successfully. It is failing when STRICT mode is set.

Will the inbound calls also will go through Envoy proxy like below and mtls will be enforced?
Consumer → Envoy → Kong Proxy → Envoy → Upstream
or
Consumer → Kong Proxy → Envoy → Upstream

How can we fix this connection errors with STRICT mtls? I tried setting upstream host header through request-transformer plugin in ingress but that also did not work.

host: helloworld.kong.svc

2

I was able to fix the issue by defining PeerAuthentication

1 Like
3

How did you solve this issue exactly?