Hi ,
I am seeing issue with kong multiple nginx master process when using kong 1.4 and 1.5 on kubenetes.
while in kong 2.0 and 1.2 issue doesnot exist , we have one master process only .
Can you elaborate on what you’re seeing, or show us what commands you’re running and their output?
I am running kong 1.4.3 inside a container and when it starts i can see multiple nginx master processed ideally it has be only one .
Can you provide what you’re using to determine that? What you describe is rather unusual, so I want to understand how you’re arriving at that conclusion. For example, if I start Kong, I get the following when running ps inside the container:
$ ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
kong 1 0.6 0.0 11844 3024 pts/0 Ss 22:35 0:00 /bin/bash
kong 51 0.1 0.0 26548 4068 pts/0 S+ 22:35 0:00 perl /usr/local/openresty/bin/resty /usr/local/bin/kong start
kong 53 1.2 0.0 56988 15212 pts/0 S+ 22:35 0:00 /usr/local/openresty/nginx/sbin/nginx -p /tmp/resty_MGmAykEyRg/ -c conf/nginx.conf
kong 71 1.8 0.1 713560 23656 pts/0 S+ 22:35 0:00 nginx: master process /usr/local/openresty/nginx/sbin/nginx -p /usr/local/kong -c nginx.conf
kong 79 0.7 0.4 768132 72196 pts/0 S+ 22:35 0:00 nginx: worker process
kong 80 0.7 0.4 768132 72196 pts/0 S+ 22:35 0:00 nginx: worker process
kong 81 0.6 0.4 768132 72312 pts/0 S+ 22:35 0:00 nginx: worker process
kong 82 0.7 0.4 768132 72332 pts/0 S+ 22:35 0:00 nginx: worker process
kong 83 11.0 0.0 11844 3032 pts/1 Ss 22:36 0:00 /bin/bash
kong 100 0.0 0.0 51772 3448 pts/1 R+ 22:36 0:00 ps aux
There shouldn’t be any way to spawn multiple master processes without some contrived configuration: the Docker image normally starts only one via its entrypoint and Kong has its own protections against running multiple instances on one machine. To spawn multiple masters inside a single container, you’d need to exec into the container and run KONG_PREFIX=/something-nonstandard kong start to start another process that doesn’t conflict with the first.
You can see multiple masters if you’re checking processes on the kubelet if multiple Kong replicas are scheduled on the same node. We don’t specify any affinity rules by default, so that’s allowed. https://github.com/Kong/charts/blob/kong-1.6.1/charts/kong/values.yaml#L407-L409 allows you to set them following something like the example at https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#never-co-located-in-the-same-node
This is the problem :
[root@kong1 /]# ps -aef | grep nginx
kong 1 0 2 23:39 ? 00:00:01 nginx : master process /usr/local/openresty/ nginx /sbin/ nginx -p /usr/local/kong -c nginx .conf
kong 281 1 23 23:39 ? 00:00:00 nginx : master process /usr/local/openresty/ nginx /sbin/ nginx -p /usr/local/kong -c nginx .conf
kong 283 1 54 23:39 ? 00:00:01 nginx : master process /usr/local/openresty/ nginx /sbin/ nginx -p /usr/local/kong -c nginx .conf
root 285 243 6 23:39 ? 00:00:00 /usr/local/openresty/ nginx /sbin/ nginx -p /tmp/resty_VNgqkeplTN/ -c conf/ nginx .conf
kong 286 1 11 23:39 ? 00:00:00 nginx : master process /usr/local/openresty/ nginx /sbin/ nginx -p /usr/local/kong -c nginx .conf
kong 287 1 15 23:40 ? 00:00:00 nginx : master process /usr/local/openresty/ nginx /sbin/ nginx -p /usr/local/kong -c nginx .conf
kong 289 1 11 23:40 ? 00:00:00 nginx : master process /usr/local/openresty/ nginx /sbin/ nginx -p /usr/local/kong -c nginx .conf
kong 291 1 0 23:40 ? 00:00:00 nginx : master process /usr/local/openresty/ nginx /sbin/ nginx -p /usr/local/kong -c nginx .conf
kong 292 1 0 23:40 ? 00:00:00 nginx : master process /usr/local/openresty/ nginx /sbin/ nginx -p /usr/local/kong -c nginx .conf
root 294 201 0 23:40 pts/0 00:00:00 grep --color=auto nginx
kong 295 1 0 23:40 ? 00:00:00 nginx : master process /usr/local/openresty/ nginx /sbin/ nginx -p /usr/local/kong -c nginx .conf
i am using image provided by kong as 1.4-centos , just running that image is causing issue .
1.4,1.5 having the issue while 2.0 no such issue seen
We’re kinda stumped on this:
kong 1 0 2 23:39 ? 00:00:01 **nginx** : master process /usr/local/openresty/ **nginx** /sbin/ **nginx** -p /usr/local/kong -c **nginx** .conf
kong 281 1 23 23:39 ? 00:00:00 **nginx** : master process /usr/local/openresty/ **nginx** /sbin/ **nginx** -p /usr/local/kong -c **nginx** .conf
kong 283 1 54 23:39 ? 00:00:01 **nginx** : master process /usr/local/openresty/ **nginx** /sbin/ **nginx** -p /usr/local/kong -c **nginx** .conf
Not only are the multiple masters present, the extra two are children of PID 1 (second column is PPID) and started at the same time.
Can you walk us through, step-by-step, how you spawn your Kong Pods? If you use a simplified means of deploying one (e.g. with kubectl run on the image alone), do you see the same behavior as in your standard deployment process?
kubectl run -it --restart=Never --rm --image kong:1.5 master-test
There are two ways to reproduce it
1.1.Kong running inside pod with version 1.2.2-centos with cassandra as backend
1.2.Upgrade deployment with new image 1.4-centos with same keyspace as step 1.1.
issue is seen .
2.1.Kong running inside pod with version 1.4-centos with new keyspace of cassandra
shall I open bug for it
May be best to–with what’s presented I cannot replicate this, e.g.
$ kubectl run -it --restart=Never --env="KONG_DATABASE=off" --rm --image kong:1.5 master-test
If you don't see a command prompt, try pressing enter.
2020/06/08 17:49:50 [notice] 1#0: signal 28 (SIGWINCH) received
2020/06/08 17:49:50 [notice] 26#0: signal 28 (SIGWINCH) received
2020/06/08 17:49:50 [notice] 27#0: signal 28 (SIGWINCH) received
2020/06/08 17:49:50 [notice] 1#0: signal 28 (SIGWINCH) received
2020/06/08 17:49:50 [notice] 26#0: signal 28 (SIGWINCH) received
2020/06/08 17:49:50 [notice] 27#0: signal 28 (SIGWINCH) received
$ kubectl exec -it master-test /bin/sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
/ $ ps -aef
PID USER TIME COMMAND
1 kong 0:00 nginx: master process /usr/local/openresty/nginx/sbin/nginx -p /usr/local/kong -c nginx.conf
26 kong 0:00 nginx: worker process
27 kong 0:00 nginx: worker process
28 kong 0:00 /bin/sh
33 kong 0:00 ps -aef
I can’t think offhand how you’d replicate that in a Dockerized environment without some very specific configuration and abnormal container management that doesn’t appear to be in place here.
It tried to run inside kubernetes with cassandra as backend .
i can help in reproduction of the issue , any particular logs required after issue is reproduced.
Can you set KONG_LOG_LEVEL=debug, start a new Pod (e.g. by scaling the Deployment replicas to 0 and then to 1), and use kubectl logs PODNAME -c proxy to retrieve the Kong container debug logs? Everything we care about should be in startup logs; waiting 30s after starting and collecting logs then should suffice.
Once you’ve done that, run the same ps command as earlier to collect process information from that run.
The ps info from earlier indicates that the additional processes were children of PID 1 (the original master), so hopefully the original master includes some log information indicating why it spawned them.
As part of testing for another issue, we’ve found a scenario that may lead to this: if a worker gets killed because it runs out of memory, it may get respawned into this weird state where it looks like or is a master.
@vinicius.mignot does that characterization match what you observed? I may be a little off, since I’m working from my memory of our call rather than seeing this symptom in practice.
@piyush_gupta what are your pod resource limits and requests, and do you see any restarts mentioned in the container log?
Yes, that’s what is happening in my tests. And if I keep pushing, all the Nginx processes become master processes.
@traines I see multiple restart on the the container , i put 1 CPU and 1Gi Memory on the pod ,shall i open the bug ticket for the issue ?
Probably check out 2.0–there was a change in 1.4.1 that removed a restriction on worker connections (see https://github.com/Kong/kong/blob/master/CHANGELOG.md#141 and https://github.com/Kong/kong/pull/5148) that we later reverted in https://github.com/Kong/kong/commit/798e270d7ed4fa3db0da76ec447676806842fa9f since it ate a bunch more memory than it should.
No–there won’t be any future 1.x releases, that change will only ever be applied to 2.x and onward.