Concurrent websocket connections

Hi,
we are load balancing websocket connections with kong.
During our load tests we recognized that kong can handle much less websocket connections than the standard kubernetes nginx.
Kong gives up at 10k concurrent connections while for nginx we could reach 60k.
Error messages of kong:

[alert] 33#0: 16384 worker_connections are not enough
[alert] 33#0: *21541620 16384 worker_connections are not enough while connecting to upstream, client: x.x.x.x, server: kong, request: "GET /cable?chatbot_name=bot HTTP/1.1", upstream: "http://10.42.64.27:8080/cable?chatbot_name=bot", host: "bot.example.com"

We tried to tune this but looking at the relevant parameters revealed kong it should be able to handle more connections than nginx.

limits kong:

worker_processes auto;   # should be 16 in our case
worker_connections 16384;
worker_rlimit_nofile 1048576;

limits nginx:

worker_processes 16;
worker_connections  16384;
worker_rlimit_nofile 64512;

Why are the limits for kong that much lower?
Any Tipps how to tune kong to get to a similar level?

https://github.com/Kong/kong/pull/5148 maybe related? This may have made it into Kong 1.4, didn’t check yet.

Edit- I see it in the Master branch, so maybe it is in 1.4 . Didn’t see it in the CHANGELOG for 1.4 though.

The PR Jeremy mentioned is not part of 1.4 and I doubt that’s the reason for this. The worker_connections seems to be same as Nginx (of course those could be increased).

I was talking to @datong.sun regarding this and he pointed out that Kong doesn’t enable reuseport1 by default.
@landor could you enable that for Kong and test?
It might be that connections are not balanced equally among the workers.

Thank you for your answers.
I will try the setting you mentioned

Tried now to set the parameter.
This indeed helps to be able to keep more concurrent connections.
Currently at 24k concurrent connections.
(because of that problem: Anycable websocket connections not closed)

Thank You


© 2019 Kong Inc.    Terms  •  Privacy  •  FAQ