I’ve decreased the number of nginx workers to 1(to decrease memory usage)
Whenever the proxy now reaches ~99% of memory usage it seems to trigger a large garbage collection that impacts incoming requests.
I lose about 500ms of traffic(~40 requests) during the GC. It would be nice if the GC triggered earlier and in a less severe manner. Is it something we can configure I wonder?
I’m currently running the proxy with 1GB of memory, maybe that is not a realistic limit.
I have done some further investigation and can see this in the log while I do some stress testing:
19-11-07 11:29:13.599 - 2019/11/07 11:29:13 [notice] 1#0: start worker process 15682 proxy
19-11-07 11:29:13.596 - 2019/11/07 11:29:13 [alert] 1#0: worker process 3572 exited on signal 9 proxy
19-11-07 11:29:13.594 - 2019/11/07 11:29:13 [notice] 1#0: signal 17 (SIGCHLD) received from 3572 proxy
19-11-07 11:29:13.583 - E1107 11:29:13.582602 1 controller.go:132] unexpected failure updating Kong configuration: ingress-controller
19-11-07 11:29:13.583 - posting new config to /config: making HTTP reqeust: Post https://localhost:8444/config?check_hash=1: EOF ingress-controller
From this time and until the next iteration for the ingress-controller to sync I get 404 for every request to an endpoind behind the api gateway. So this indicates that the dbless config is somehow wiped from the memory when the worker process is killed