Deadline Execeeded error when gRPC server graceful restart

Kubernetes
1

I have a problem. When i rollout restart the grpc server and when the old pod destory, the client will report grpc deadline execeeded error. But i use server.GracefulStop() in server side and use k8s svc directly, it nothing happened.

version:

kong: 2.3.1
kong-ingress-controller: 1.1.0

kong error log

kong-6c684f8fc5-29mdf kong 2021/04/01 10:36:59 [warn] 28#0: *26699180 [lua] ring.lua:246: redistributeIndices(): [upstream:uranus-alpha-grpc.zaihui-platform.9000.svc 183265] redistributed indices, size=10000, dropped=10000, assigned=0, left unassigned=10000, context: ngx.timer

kong-6c684f8fc5-5z7md kong 2021/04/01 10:37:01 [warn] 27#0: *26762659 a client request body is buffered to a temporary file /usr/local/kong/client_body_temp/0000000573, client: 172.19.25.197, server: kong, request: "POST /zaihui.base.BaseService/GetTime HTTP/2.0", host: "host:9000"

kong-6c684f8fc5-5z7md kong 2021/04/01 10:37:09 [error] 27#0: *26762659 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.19.25.197, server: kong, request: "POST /zaihui.base.BaseService/GetTime HTTP/2.0", upstream: "grpc://192.168.36.105:6000", host: "host:9000"

ingress.yml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: grpc
  namespace: ${NAMESPACE}
  annotations:
    konghq.com/protocols: "grpc,grpcs"
spec:
  rules:
    - host: domain
      http:
        paths:
          - path: /
            backend:
              serviceName: -grpc
              servicePort: 9000

kong custom config

  - KONG_ANONYMOUS_REPORTS=off
  - KONG_HTTP2=on
  - KONG_PROXY_LISTEN="0.0.0.0:8000, 0.0.0.0:9000 http2"
  - KONG_ADMIN_LISTEN="0.0.0.0:8001"
  - KONG_STATUS_LISTEN="0.0.0.0:9254"
  - KONG_PLUGINSERVER_NAMES=go-auth
  - KONG_PLUGINSERVER_GO_AUTH_SOCKET=/usr/local/kong/go-auth.socket
  - KONG_PLUGINSERVER_GO_AUTH_START_CMD="/usr/local/bin/go-auth"
  - KONG_PLUGINSERVER_GO_AUTH_QUERY_CMD="/usr/local/bin/go-auth -dump"
  - KONG_UNTRUSTED_LUA=on

  - KONG_ADMIN_ACCESS_LOG=/dev/null
  - KONG_ADMIN_ERROR_LOG=/dev/null
  - KONG_PROXY_ACCESS_LOG=/dev/stdout main
  - KONG_PROXY_ERROR_LOG=/dev/stderr
  - KONG_TRUSTED_IPS=0.0.0.0/0
  - KONG_REAL_IP_HEADER=X-Forwarded-For
  - KONG_NGINX_ADMIN_CLIENT_MAX_BODY_SIZE=100m
  - KONG_ERROR_DEFAULT_TYPE=application/json

  - KONG_DATABASE=postgres
  - KONG_NGINX_PROXY_INCLUDE=/kong/proxy.nginx.conf

Thanks.

2

Hi @Ackerr :wave:

I just want to double check that I understand your problem correctly, can you please let me know if the following statements are true:

  1. you have a custom gRPC service which you are configuring with Kong Ingress
  2. when pods are killed, some connections remain and time out

If the above are true, can you confirm whether this happens on persistent/existing connections only, or are new connections hitting the timeout as well?