Hi all, We have 3 Kong replicas deployed in Kubernetes from stable/kong 0.19.1 Helm chart). We have upgraded Kong to 1.4 using the alpine images. Kong is configured in DB-less mode. We have checked that when the edge load balancer points to a particular replica, for a particular route, the service…

Interesting. Do you see any error in the controller of the two pods that were not able to serve requests?

Nothing! Currently one of them has been deleted and Kong retrieves 503 (Right behavior) and 504 alternatively. We can access directly with port-forwarding without any issue. Attached a recent evidence made with Apachebench. Kong pods “-7rw…” and “-2cv…” retrieve 504, “-dnx…” retrieves 503. [41] I…

What is your k8s environment and version? The issue doesn’t seem to be with Kong itself but with Kubernetes or the load-balancer in-front of Kong. Kong is responding with 499, meaning the load balancer closed the connection before Kong could send back the response. Some things to ensure: Is thi…

Thank you, Each replica (3) is deployed in a different node. Currently there is no target upstream, it is not deployed. Load balancer discarded, I have just tried requesting directly to each pod (with port forwarding) using curl and only “-dnx” is responding properly. Kubernetes version is 1.11 We…

Do you mean that only two services that are being proxies by Kong have this issue or is it two Kong pods that have this issue? After LB is removed, what error do you get back from Kong when you curl it directly over the port-forwarding tunnel?

Yes, only these two services have the issue, the others work perfectly. I have repeated the test asking resources to Jenkins directly to a Kong pod (without ELB) and this is the result: > > 2019/11/21 21:42:46 [error] 36#0: *182367757 [lua] init.lua:800: balancer(): failed to retry the dns/balanc…

Can you share the Ingress resource for Jenkins? By any chance, do you have the ingress.kubernetes.io/service-upstream annotation applied anywhere?

Yes! apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: configuration.konghq.com: jenkins-ingress kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"kong"}…

Okay. Nothing interesting here. Kong should not be doing a DNS lookup for jenkins.ci.svc hostname. That should be coming form Upstreams. Do you see an upstream in Kong’s ADmin API associated with jenkins.ci.svc (and also targets)?

Yes! upstream and targets are discovered correctly by Kong. However we get the same issue with and without targets. First picture (Nginx traces) In this topic shows 200 responses and 499 when pod is running and target discovered. Third picture, also Nginx traces, shows 503 response and 499 when po…

Kong 1.4, K8S, DB-less, 504 Gateway timeout

General

abenitovsc November 27, 2019, 11:47am 14

Following the changes described in this topic, I have applied them and I will response with the feedback.

Topic		Replies	Views
DNS resolution failed in kong ingress pod Questions	4	5962	January 31, 2020
Kong v 1.4.0 - Getting 504s when adding plugin to route via REST (SOLVED) Questions	9	2050	January 15, 2020
Timeout in connection to upstream Questions kubernetes	4	6607	December 5, 2019
Kong random 404 after worker restart General	0	679	August 2, 2022
Kong 1.4.0 - Getting random 503 status codes on requests Questions kubernetes	9	5912	July 18, 2020

Kong 1.4, K8S, DB-less, 504 Gateway timeout

Related topics