Kong + Istio + gRPC: server closed the stream without sending trailers

Hi, I have been trying to get Kong working alongside Istio for routing to a gRPC service (Hello World). For context, I am using the official Kong helm chart at 1.3.0 and Istio 1.4.3. Based on the Kong/Istio blogpost, it seems like I would simply need to enable sidecar injection for the Kong proxy deployment. Kong works and routes correctly to the gRPC service without the sidecar, however, once it is injected, I see the error “rpc error: code = Internal desc = server closed the stream without sending trailers”.

Some other details of my setup:

Some findings:

  • There are no errors on the gRPC server-side
  • I see a log entry ’ [02/Mar/2020:23:12:32 +0000] “POST /helloworld.Greeter/SayHello HTTP/2.0” 200 18 “-” “grpc-go/1.26.0” ’ which seems to imply the request/response was ok
  • Looking at debug level logs in the kong envoy sidecar, I see this entry: ’ [Envoy (Epoch 0)] [2020-03-02 05:42:37.055][26][debug][pool] [external/envoy/source/common/http/http1/conn_pool.cc:206] [C102] response complete ’ which suggests an http1 connection was used. I do not see this log on the server-side envoy sidecar nor when I call the server from another pod within the mesh
  • Enabling PILOT_ENABLE_PROTOCOL_SNIFFING_FOR_INBOUND in Istio Pilot and renaming the service port name to something random (so that it default to TCP) appears to get rid of this error, but Istio routing rules would no longer apply in this case.

It seems like the issue is that the http2 connection is being downgraded to http1 in the Kong envoy sidecar. Is there some reason for this or some configuration I am missing? Happy to provide more info if needed. I have poked around quite a bit with no success.

I’m not a ninja with Istio and Envoy so what I say here might be wrong but this is how I would go about:

  1. TCPdump the traffic coming out of Kong and make sure that the HTTP host header is set correctly so that envoy can detect and route traffic correctly.
  2. Figure out why envoy is downgrading the connection.

@hbagdi Thanks for the reply and your first suggestion ultimately leads to the issue. After digesting this thread: Kong Ingress controller is unable to route request to Istio virtual services it seems the issue was that both the hostname and service port must match what is used to dial Kong.

1 Like

© 2019 Kong Inc.    Terms  •  Privacy  •  FAQ