104: Connection reset by peer while reading response header from upstream

Questions
2

Hi @donalddw,

Welcome to the fabulous world of reverse-proxies! “Connection reset by peer” should probably be considered as the initiation rite of any Konger/NGINX user: it isn’t an uncommon error at all. What it means (you might be familiar with it but future readers might not) is that the remote server (your upstream) hung up on Kong and sent an RST packet without bothering doing the usual FIN-ACK handshake. Quite rude.

As the error is saying, it seems like the remote is closing its connection while Kong is receiving the response headers - you already found that out by disabling HTTPs, which didn’t help troubleshoot the issue.

You might also want to:

  • investigate your remote server’s logs to find out if any error occurred at that time that may have caused it to react so abruptly.
  • capture the traffic between Kong and the remote server via tools such as tcpdump or Wireshark to gather information and be able to better draw hypothesis as to the root cause, potentially even replicate the issue without Kong, if you find out what is triggering it.
  • restart Kong with log_level = debug and keep an eye on the logs as such errors occurs. (you already did so, but future readers of this issue may not)
  • ensure that your remote is accepting HTTP/1.1 traffic (vs 1.0).
  • as you suggested, try to tweak the ngx_http_proxy_module directives such that it sends Connection: close by default (although its interpretation is up to your remote anyway - and reminder: in HTTP/1.1, all connections are considered kept-alive unless Connection: close is explicitly set). Another possible tweak is updating the proxy_http_version to 1.0 in case the above turns out to be a requirement. The only way to tweak those settings is by using a Custom NGINX configuration template.

I think those steps already amount to a good/standard way of debugging this issue. If I can think of anything else, I’ll come back and update this list. I’m hopping that since this error is particularly common, your topic will come out when users try to dig for answers before posting their own question.

Good hunting and let us know of your findings!

Best,

4 Likes