Hey, thanks for the reply!
Actually, I can see the traffic hitting productpage’s Envoy sidecar. It definitely looks like the traffic is making it to the right destination, but the client sidecar (Kong’s Envoy proxy) seems to be sending a request the server Envoy doesn’t like.
[2020-04-07 23:56:54.672][31][debug][filter] [external/envoy/source/extensions/filters/listener/original_dst/original_dst.cc:18] original_dst: New connection accepted
[2020-04-07 23:56:54.672][31][debug][filter] [external/envoy/source/extensions/filters/listener/tls_inspector/tls_inspector.cc:72] tls inspector: new connection accepted
[2020-04-07 23:56:54.672][31][debug][main] [external/envoy/source/server/connection_handler_impl.cc:287] [C29] new connection
[2020-04-07 23:56:54.672][31][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:167] [C29] handshake error: 1
[2020-04-07 23:56:54.672][31][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:200] [C29] TLS error: 268435612:SSL routines:OPENSSL_internal:HTTP_REQUEST
[2020-04-07 23:56:54.672][31][debug][connection] [external/envoy/source/common/network/connection_impl.cc:190] [C29] closing socket: 0
As for the host header, the blog does specify that needs to be dealt with, and I have that config in place:
apiVersion: configuration.konghq.com/v1
kind: KongIngress
metadata:
name: do-not-preserve-host
namespace: default
route:
preserve_host: false
For reference, I have mTLS globally enabled with MeshPolicy and a DestinationRule, and istioctl says that my mTLS config is bueno.
MeshPolicy:
apiVersion: authentication.istio.io/v1alpha1
kind: MeshPolicy
metadata:
name: default
spec:
peers:
- mtls: {}
And the Destination Rule:
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
labels:
app: security
chart: security
heritage: Helm
release: istio
name: default
namespace: istio-system
spec:
host: '*.local'
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
And istioctl authn tls-check:
:~/istio-bin/istio-1.3.3$ istioctl authn tls-check ingress-kong-7c955554f6-2qrj7 -n kong
HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
details.default.svc.cluster.local:9080 OK mTLS mTLS default/ default/istio-system
istio-citadel.istio-system.svc.cluster.local:8060 OK mTLS mTLS default/ default/istio-system
istio-citadel.istio-system.svc.cluster.local:15014 OK mTLS mTLS default/ default/istio-system
istio-galley.istio-system.svc.cluster.local:443 OK mTLS mTLS default/ default/istio-system
istio-galley.istio-system.svc.cluster.local:9901 OK mTLS mTLS default/ default/istio-system
istio-galley.istio-system.svc.cluster.local:15014 OK mTLS mTLS default/ default/istio-system
istio-ingressgateway.istio-system.svc.cluster.local:80 OK mTLS mTLS default/ default/istio-system
istio-ingressgateway.istio-system.svc.cluster.local:443 OK mTLS mTLS default/ default/istio-system
istio-ingressgateway.istio-system.svc.cluster.local:15020 OK mTLS mTLS default/ default/istio-system
istio-ingressgateway.istio-system.svc.cluster.local:15029 OK mTLS mTLS default/ default/istio-system
istio-ingressgateway.istio-system.svc.cluster.local:15030 OK mTLS mTLS default/ default/istio-system
istio-ingressgateway.istio-system.svc.cluster.local:15031 OK mTLS mTLS default/ default/istio-system
istio-ingressgateway.istio-system.svc.cluster.local:15032 OK mTLS mTLS default/ default/istio-system
istio-ingressgateway.istio-system.svc.cluster.local:15443 OK mTLS mTLS default/ default/istio-system
istio-ingressgateway.istio-system.svc.cluster.local:31400 OK mTLS mTLS default/ default/istio-system
istio-pilot.istio-system.svc.cluster.local:8080 OK mTLS mTLS default/ default/istio-system
istio-pilot.istio-system.svc.cluster.local:15010 OK mTLS mTLS default/ default/istio-system
istio-pilot.istio-system.svc.cluster.local:15011 OK mTLS mTLS default/ default/istio-system
istio-pilot.istio-system.svc.cluster.local:15014 OK mTLS mTLS default/ default/istio-system
istio-policy.istio-system.svc.cluster.local:9091 CONFLICT mTLS HTTP default/ istio-policy/istio-system
istio-policy.istio-system.svc.cluster.local:15004 CONFLICT mTLS HTTP default/ istio-policy/istio-system
istio-policy.istio-system.svc.cluster.local:15014 CONFLICT mTLS HTTP default/ istio-policy/istio-system
istio-sidecar-injector.istio-system.svc.cluster.local:443 OK mTLS mTLS default/ default/istio-system
istio-sidecar-injector.istio-system.svc.cluster.local:15014 OK mTLS mTLS default/ default/istio-system
istio-telemetry.istio-system.svc.cluster.local:9091 CONFLICT mTLS HTTP default/ istio-telemetry/istio-system
istio-telemetry.istio-system.svc.cluster.local:15004 CONFLICT mTLS HTTP default/ istio-telemetry/istio-system
istio-telemetry.istio-system.svc.cluster.local:15014 CONFLICT mTLS HTTP default/ istio-telemetry/istio-system
istio-telemetry.istio-system.svc.cluster.local:42422 CONFLICT mTLS HTTP default/ istio-telemetry/istio-system
kong-proxy.kong.svc.cluster.local:80 OK mTLS mTLS default/ default/istio-system
kong-proxy.kong.svc.cluster.local:443 OK mTLS mTLS default/ default/istio-system
kong-validation-webhook.kong.svc.cluster.local:443 OK mTLS mTLS default/ default/istio-system
kube-dns.kube-system.svc.cluster.local:53 OK mTLS mTLS default/ default/istio-system
kube-dns.kube-system.svc.cluster.local:53 OK mTLS mTLS default/ default/istio-system
kubernetes.default.svc.cluster.local:443 CONFLICT mTLS HTTP default/ api-server/istio-system
productpage.default.svc.cluster.local:9080 OK mTLS mTLS default/ default/istio-system
prometheus.istio-system.svc.cluster.local:9090 OK mTLS mTLS default/ default/istio-system
ratings.default.svc.cluster.local:9080 OK mTLS mTLS default/ default/istio-system
reviews.default.svc.cluster.local:9080 OK mTLS mTLS default/ default/istio-system
I’ve torn this Kong + Istio setup down and rebuilt it twice to make sure I didn’t do something silly, and I got the same result both times. As soon as I change the MeshPolicy to mtls: PERMISSIVE, the traffic goes through. It doesn’t make sense to me, as there are no other Destination Rules in play, it’s a fresh install, and the mTLS config looks right. But here we are.
It would be really helpful if someone else could test and confirm/disprove what I’m seeing! I’m also open to more troubleshooting suggestions.