Unable to route request to Istio virtual services

Questions
1

Hello, is there anybody who can help with this issue? I’m facing an issue with routing to Istio virtual services…

My Current setup:

My findings so far:

  1. Requests coming out from the client > kong > my-service is not intercepted by the proxy even though the host is equal to my-service.
  2. Requests coming out from kong (curl from kong container) > service is intercepted by the proxy

and here are the TCP dump result from both requests consecutively:

  1. Client > Kong > Service
Frame 703: 584 bytes on wire (4672 bits), 584 bytes captured (4672 bits)
Ethernet II, Src: ea:72:a9:ea:9b:e6 (ea:72:a9:ea:9b:e6), Dst: aa:ff:0f:31:aa:85 (aa:ff:0f:31:aa:85)
Internet Protocol Version 4, Src: 10.86.3.78 (10.86.3.78), Dst: 3964386632353831.dev-global-aaa.dev.svc.cluster.local (10.85.180.38)
Transmission Control Protocol, Src Port: 58448 (58448), Dst Port: http (80), Seq: 1, Ack: 1, Len: 518
Hypertext Transfer Protocol
    GET /login HTTP/1.1\r\n
    Host: dev-global-aaa.dev.svc\r\n
    Connection: keep-alive\r\n
    X-Forwarded-For: 103.6.151.209\r\n
    X-Forwarded-Proto: https\r\n
    X-Forwarded-Host: api-dev.example.com\r\n
    X-Forwarded-Port: 8443\r\n
    X-Real-IP: 103.6.151.209\r\n
    user-agent: curl/7.64.1\r\n
    accept: */*\r\n
    X-Consumer-ID: 864ed4cf-a28c-4bec-92b7-f0f760750ec6\r\n
    X-Consumer-Custom-ID: anonymous-user\r\n
    X-Consumer-Username: anonymous-user\r\n
    X-Anonymous-Consumer: true\r\n
    nvcountry: global\r\n
    x-nv-system-id: global\r\n
    x-nv-request-uuid: 4d869056-ffdb-4f91-8285-eaae713c88d9\r\n
    \r\n
    [Full request URI: http://dev-global-aaa.dev.svc/login]
    [HTTP request 1/5]
    [Response in frame: 705]
    [Next request in frame: 985]
  1. Kong > Service
Ethernet II, Src: ea:72:a9:ea:9b:e6 (ea:72:a9:ea:9b:e6), Dst: aa:ff:0f:31:aa:85 (aa:ff:0f:31:aa:85)
Internet Protocol Version 4, Src: 10.86.3.78 (10.86.3.78), Dst: 10.86.24.45 (10.86.24.45)
Transmission Control Protocol, Src Port: 38098 (38098), Dst Port: cslistener (9000), Seq: 1, Ack: 1, Len: 1615
Hypertext Transfer Protocol
    GET /login HTTP/1.1\r\n
    host: dev-global-aaa.dev.svc\r\n
    user-agent: curl/7.66.0\r\n
    accept: */*\r\n
    x-forwarded-proto: http\r\n
    x-request-id: 1469023f-95ef-4955-8022-e8a7767c80de\r\n
    x-envoy-decorator-operation: dev-global-aaa.dev.svc.cluster.local:80/*\r\n
     [truncated]x-envoy-peer-metadata: ChwKDElOU1RBTkNFX0lQUxIMGgoxMC44Ni4zLjc4CrsCCgZMQUJFTFMSsAIqrQIKDQoDYXBwEgYaBGtvbmcKDAoDZW52EgUaA2RldgofCgRuYW1lEhcaFWRldi1nbG9iYWwta29uZy0xLTUtMQohChFwb2QtdGVtcGxhdGUtaGFzaBIMGgo1NGI1Y2JkZjVkCh4KB3JlbGV
    x-envoy-peer-metadata-id: sidecar~10.86.3.78~dev-global-kong-1-5-1-54b5cbdf5d-h42pr.dev~dev.svc.cluster.local\r\n
    x-b3-traceid: e500b44194677918e2736bcf37ccdb2d\r\n
    x-b3-spanid: e2736bcf37ccdb2d\r\n
    x-b3-sampled: 0\r\n
    content-length: 0\r\n
    \r\n
    [Full request URI: http://dev-global-aaa.dev.svc/login]
    [HTTP request 1/3]
    [Response in frame: 65]
    [Next request in frame: 419]

And my suspicion is that kong with the service-upstream annotation sends correctly to the service but using other hostname which resolves to correct service ip but is not in the istio whitelist rule as shown as in the following figure.

image
image1265×165 31.5 KB

1 Like
3

For everybody who might have encountered similar issues:

  1. Make sure that preserve_host is set to false
  2. Change the kong user id from 1337 since it’s clashing with istio-proxy user
1 Like
4

I somehow missed this post. Glad to see that this is working for you now. Thanks!

This should not be the case anymore with Kong 2.0.

1 Like
5

I also have the same problem on istio service mesh.

There two solutions work for me

  1. Set preserve_host to false, it works for me
  2. Add external FQDN (which client curl to ) to Istio VirtualService spec.hosts[]

Thanks for your help.