Concurrent websocket connections

Hi,
we are load balancing websocket connections with kong.
During our load tests we recognized that kong can handle much less websocket connections than the standard kubernetes nginx.
Kong gives up at 10k concurrent connections while for nginx we could reach 60k.
Error messages of kong:

[alert] 33#0: 16384 worker_connections are not enough
[alert] 33#0: *21541620 16384 worker_connections are not enough while connecting to upstream, client: x.x.x.x, server: kong, request: "GET /cable?chatbot_name=bot HTTP/1.1", upstream: "http://10.42.64.27:8080/cable?chatbot_name=bot", host: "bot.example.com"

We tried to tune this but looking at the relevant parameters revealed kong it should be able to handle more connections than nginx.

limits kong:

worker_processes auto;   # should be 16 in our case
worker_connections 16384;
worker_rlimit_nofile 1048576;

limits nginx:

worker_processes 16;
worker_connections  16384;
worker_rlimit_nofile 64512;

Why are the limits for kong that much lower?
Any Tipps how to tune kong to get to a similar level?

https://github.com/Kong/kong/pull/5148 maybe related? This may have made it into Kong 1.4, didn’t check yet.

Edit- I see it in the Master branch, so maybe it is in 1.4 . Didn’t see it in the CHANGELOG for 1.4 though.

The PR Jeremy mentioned is not part of 1.4 and I doubt that’s the reason for this. The worker_connections seems to be same as Nginx (of course those could be increased).

I was talking to @datong.sun regarding this and he pointed out that Kong doesn’t enable reuseport1 by default.
@landor could you enable that for Kong and test?
It might be that connections are not balanced equally among the workers.

Thank you for your answers.
I will try the setting you mentioned

Tried now to set the parameter.
This indeed helps to be able to keep more concurrent connections.
Currently at 24k concurrent connections.
(because of that problem: Anycable websocket connections not closed)

Thank You

Hello again,

we are facing that problem again since we upgraded the kong helm chart from 0.21.1 to 1.0.0. Kong version ist still 1.3.1 at the moment. Will upgrade tonight to 1.4.3 and report back.

Any idea why this could have changed with just the helm chart update?
Both versions end up having it in /kong_prefix/nginx-kong.conf

Thanks

Helm chart upgrade shouldn’t cause any change.
Can you compare the environment variables set by 0.21.1 and that by 1.0.0 and report here what has changed? That will point out the reason for the change.

I’d recommend not upgrading Kong and the Helm Chart at the same time, as it then makes it very difficult to figure out what is the root cause of the problem.

Hi " landor", we are using a use case where we are looking for “websocket” connection with “KONG API GATEWAY” exactly, how you did. we are using AKS as infrastructure.

Can you kindly assist me on same, how to that.

Thanks & Regards
Jaiswar Vipin Kumar R.