Kong + Istio + STRICT mTLS (503 errors)

1

We are attempting to setup the following scenario: Kong + Istio + STRICT mTLS (via PeerAuthentication)

image
image1491×466 26.2 KB

Our aim:

  1. Users connect via http/https to the AKS cluster exposed via the Azure/k8s LB using hostname. in our case one on .cloudapps.com
  2. Kong receives the request and as it is deployed with istio sidecar injected and makes request using mTLS to the upstream service.

Issue:
When PeerAuthentication is added and STRICT mode enabled then we receive:
upstream connect error or disconnect/reset before headers. reset reason: connection termination

Here’s Kiali with PERMISSIVE - Working as expected

image
image615×591 35.1 KB

Here’s Kiali with STRICT (KongIngress - do-not-preserve-host, Ingress with annotation for do-not-preserve-host and productpage svc without upstream annotation). As you can see the Traffic never hits the productpage side of things but just bounces around.

image
image597×579 35.2 KB

proxy container logs from ingress-kong POD:
10.240.0.4 - - [12/Feb/2021:12:58:28 +0000] "GET /favicon.ico HTTP/1.1" 503 95 "http://***.ukwest.cloudapp.azure.com/productpage?u=test" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36"

istio-proxy container logs from ingress-kong POD:
[2021-02-12T12:58:46.292Z] "GET /productpage?u=test HTTP/1.1" 503 UC upstream_reset_before_response_started{connection_termination} - "-" 0 95 0 - "10.240.0.4" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15" "2f61f602-493b-4b3f-b60f-9bb5eb3b7f5c" "***.ukwest.cloudapp.azure.com" "10.0.63.69:9080" PassthroughCluster 10.240.0.69:40822 10.0.63.69:9080 10.240.0.4:0 - allow_any

istio-proxy container logs from productpage POD:
[2021-02-12T12:58:46.293Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 10.240.0.100:9080 10.240.0.69:40822 - -

We have followed the guide provided by KONG:

@hbagdi are there plans to update the docs with an mTLS scenario?, I’ve followed along with the various threads that have been mentioned here and in GitHub but haven’t been able to get any of them running.

Configuration:
Kong deployed into kong namespace with istio-injection: enabled
Apps deployed into apps namespace with istio-injection: enabled

Peer Authentication:

apiVersion: security.istio.io/v1beta1

kind: PeerAuthentication

metadata:

  name: default

  namespace: kong

spec:

  mtls:

    mode: STRICT

---

apiVersion: security.istio.io/v1beta1

kind: PeerAuthentication

metadata:

  name: default

  namespace: apps

spec:

  mtls:

    mode: STRICT

Ingress:

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

  name: productpage

  namespace: apps

  annotations:

    kubernetes.io/ingress.class: kong

    configuration.konghq.com: do-not-preserve-host

spec:

  rules:

  - http:

      paths:

      - path: /

        backend:

          serviceName: productpage

          servicePort: 9080

KongIngress:
apiVersion: configuration.konghq.com/v1

kind: KongIngress

metadata:

  name: do-not-preserve-host

  namespace: apps

route:

  preserve_host: false

upstream:

  host_header: productpage.apps.svc

Service:
apiVersion: v1 kind: Service metadata: annotations: ingress.kubernetes.io/service-upstream: "true" labels: app: productpage service: productpage name: productpage namespace: apps spec: clusterIP: 10.0.63.69 ports: - name: http port: 9080 protocol: TCP targetPort: 9080 selector: app: productpage sessionAffinity: None type: ClusterIP

2

We were able to isolate the issue,

configuration.konghq.com: do-not-preserve-host

Should be

konghq.com/override: do-not-preserve-host

We also found no discernible difference between setting the upstream header and not setting it.