Connection and Upgrade Headers modified

Kubernetes
1

Hi,
I’m trying to run JupyterHub behind Kong in Kubernetes. Most things work fine, but Terminals are unresponsive to user input.
When I log the user pod, I am seeing 400’s pop up on websocket connections:
400 GET /user/{username}/terminals/websocket/1
This is definitely a Kong issue, because if I port-forward the service directly everything works fine.
I tried replicating the request with CURL using the same headers as the browser passes, and I got the response: Can "Upgrade" only to "WebSocket"
This is something which tornado does when the upgrade header does not equal websocket, so I pointed the same request at an echo-server to see what headers the server is recieving. Indeed, the upgrade header is gone entirely, and the connection header was replaced:

CURL:
> GET / HTTP/1.1
> Connection: keep-alive, Upgrade
> Upgrade: websocket

Echo:
Request Headers:
	connection=keep-alive

How can I prevent Kong from overwriting these headers? The documentation says these headers should not be overwritten anyway?

2

Are you indeed on 1.4? That version had a bug that mangles Connection improperly with Firefox-style values (keep-alive, Upgrade instead of Upgrade alone). https://github.com/Kong/kong/commit/f1f1c61333f908c327e31f77c6c478a1902f0b2b fixes it as of 2.0.1. It doesn’t look like we released any 1.x version with it; you’d need to patch the image manually if you don’t want to upgrade yet.

1 Like
3

We’ve been running JupityrHub behind Kong, and I’m just confirming traines idea, since we ran into that exact bug.

4

Are you indeed on 1.4?

Yes I am, but I’m unsure if that bug is the culprit. I have the same behavior on Chrome

5

Tried upgrading to Kong 2.2.1, but identical behavior. Terminal is unresponsive and websocket returns 400:

Request headers:
GET /user/dummy/terminals/websocket/1 HTTP/1.1
Host: jupyter.<company>.<com>
User-Agent: Mozilla/5.0 (<User OS>) Gecko/20100101 Firefox/84.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Sec-WebSocket-Version: 13
Origin: https://jupyter.<company>.<com>
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: <key>
Connection: keep-alive, Upgrade
Cookie: jupyterhub-user-dummy=<cookie>
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket

Response Headers:
HTTP/1.1 400 Bad Request
content-security-policy: frame-ancestors 'self';report-uri /hub/security/csp-report
Content-Type: text/html; charset=UTF-8
date: Thu, 07 Jan 2021 09:30:00 GMT
server: TornadoServer/6.1
Via: kong/2.2.1
x-content-type-options: nosniff
x-jupyterhub-version: 1.2.2
X-Kong-Proxy-Latency: 1
X-Kong-Upstream-Latency: 3
Content-Length: 34
Connection: keep-alive

I don’t have an echoserver set up to check the upstream headers on the cluster I’m testing on, but presume the same thing is happening with Connection being overridden and Upgrade being removed.

@tyree731 Did you ever solve this?

6

Since I believe you can run custom scripts on the JupityrHub side of things, maybe try printing out the request body JupityrHub is receiving and see if perhaps something is getting lost in translation.

Also, what is your route and service configuration? Are you running any plugins that would affect the request?

7

What are you seeing with an echo-server now? Is there possibly any other HTTP-aware hop between Kong and JupyterHub?

8

This is a clean install of Kong, no KongPlugins or KongIngresses configured. My Kubernetes Ingress is created by the JupterHub helm chart. The corresponding values are:

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: kong
  hosts:
  - jupyter.newcluster.<company>.<com>

I’ll do an echo test and post that soon

9

Request headers:
GET / HTTP/1.1
Host: echo.newcluster.company.com
User-Agent: curl/7.54.0
Accept: /
Connection: keep-alive, Upgrade
Upgrade: websocket

Upstream headers:
accept=/
connection=keep-alive
host=echo.newcluster.company.com
user-agent=curl/7.54.0
x-forwarded-for=IP, IP
x-forwarded-host=echo.newcluster.company.com
x-forwarded-path=/
x-forwarded-port=443
x-forwarded-proto=https
x-real-ip=IP

There is no hop between Kong and Jupyterhub, but kong is itself behind an AWS Classic Loadbalancer

10

You’re using a classic load balancer? Can you try using an Application Load Balancer? I know ALB’s support websockets, but I’m not so sure classic load balancers do.