502 & Keepalive connection closed, AWS/CloudFront + classic LB + Kong (K8s)

Hi Kong community,

We are facing some very annoying issue. For few requests (a day) we are getting 502 responses.
Most of the requests are fine (2xx), but for those few it is not working.

Issue is not related to request itself or the source of request. As the request rerun is fine and for the same source we got 2xx as well.

CloudFront is just saying: 502, AbortedOrigin. Logs from AWS:
upstream prematurely closed connection while reading response header from upstream

We have custom plugin, at the very begin of the plugin we added log entry to check if failing request is hitting Kong, and we do not see log entry for given request id.

The only message in Kong logs we see (when debug mode is turned on): keepalive connection closed.

We found issue related to Nginx timeout & AWS LB iddle connection when have the same value. Change to different values did not help.

Kong 2.0.4

How can I disable keepalive functionality in Kong?

How can I disable keepalive functionality in Kong?

It’s not clear to me from the description whether this will solve your problem, but to your specific question, here’s how to disable the upstream keepalive — note however that this is likely to have a performance impact:

Depending on your upstream’s behavior, you may want to tune the other related options upstream_keepalive_max_requests and upstream_keepalive_idle_timeout (also in the link above).

Hi Hisham,

Thanks for you reply.

In general, I think I found (more or less) the root cause.
Here: https://blog.percy.io/tuning-nginx-behind-google-cloud-platform-http-s-load-balancer-305982ddb340 in 3rd point there is described similar issue.
Pretty the same symptoms:

  • only POST requests
  • only few requests
  • Load Balancer in front of Nginx
    The only difference is, he is using Google LB (not AWS LB).

I was trying to turn off keepalive using provided documentation (both 2.1.x and 2.0.x).
But when I make a call:
curl -Iv http://localhost:8443/myendpoint
I still see: Connection #0 to host 192.168.64.60 left intact which suggest that keepalive is turned on.
But using ab I see like it is turned off:

(..)
Complete requests:      500
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      443006 bytes
HTML transferred:       340000 bytes
(..)

Turning off keepalive did not solve the original issue :frowning:

Are you using upstreams and targets from Kong to load balance your requests?

An HTTP 502 error means that a server received an invalid response from another server. In this case, Kong received an invalid response from your upstream. It is all explained in the log really: upstream prematurely closed connection while reading response headers. Your upstream prematurely closed the connection while Kong was patiently expecting its response.

In order to resolve this, you will have to investigate this from the upstream, see if any errors occurred there (since the issue arises with POST requests, is there a hard limit on payload sizes being met somewhere maybe?), and if not, then capturing the traffic to understand exactly what is going on will be required. This is similarly investigated as an upstream connection reset error for example.

@thibaultcha thank you for your reply.

However, we are pretty sure that the request did not hit the upstream at all.
We have a custom Kong plugin (runs before hitting upstream) where we log
every request id. We were not able to see any request id of failing requests.
(But we were able to see request id of successful requests).

Retry of the same (failing) request is successful. Which convinced us that
issue is not related to request itself.