The reason i choose “kong reload” to update config is, when use “http :8001/config config=@config.yaml”, it will time out after a while, and so many errors in error.log
http :8001/config config=@300-kong.yaml
http: error: Request timed out (30s).
My server got 32 core, so there are 32 workers, for benchmark purpose, I set 300 targets for the upstream.
then I use reload to update and got another problem: github issue link: https://github.com/Kong/kong/issues/4808
error.log:
2019/07/18 14:34:18 [notice] 10975#0: *6122 [lua] cache.lua:317: purge(): [DB cache] purging (local) cache, client: 127.0.0.1, server: kong_admin, request: “POST /config HTTP/1.1”, host: “localhost:8001”
balancer not found for testsant, context: ngx.timer
2019/07/18 14:34:24 [error] 10951#0: *6151 [lua] balancer.lua:592: on_target_event(): target create: balancer not found for testsant, context: ngx.timer
2019/07/18 14:34:24 [error] 10951#0: *6151 [lua] balancer.lua:592: on_target_event(): target create: balancer not found for testsant, context: ngx.timer
2019/07/18 14:34:24 [error] 10951#0: *6151 [lua] balancer.lua:592: on_target_event(): target create: balancer not found for testsant, context: ngx.timer
2019/07/18 14:34:24 [error] 10951#0: *6151 [lua] balancer.lua:592: on_target_event(): target create: balancer not found for testsant, context: ngx.timer
2019/07/18 14:34:24 [error] 10951#0: *6151 [lua] balancer.lua:592: on_target_event(): target create: balancer not found for testsant, context: ngx.timer
2019/07/18 14:34:24 [error] 10951#0: *6151 [lua] balancer.lua:592: on_target_event(): target create: balancer not found for testsant, context: ngx.timer
2019/07/18 14:34:24 [error] 10951#0: *6151 [lua] balancer.lua:592: on_target_event(): target create: balancer not found for testsant, context: ngx.timer
2019/07/18 14:34:24 [error] 10954#0: *6144 [lua] balancer.lua:200: populate_healthchecker(): [healthchecks] failed adding target: failed to acquire lock: timeout, context: ngx.timer
…
…
2019/07/18 14:34:24 [error] 10965#0: *6127 [lua] events.lua:175: do_handlerlist(): worker-events: event callback failed; source=balancer, event=upstreams, pid=10975 error=’/usr/local/share/lua/5.1/resty/healthcheck.lua:390: attempt to index field ‘targets’ (a nil value)
stack traceback:
/usr/local/share/lua/5.1/resty/healthcheck.lua:390: in function ‘get_target_status’
/usr/local/share/lua/5.1/kong/runloop/balancer.lua:196: in function ‘populate_healthchecker’
/usr/local/share/lua/5.1/kong/runloop/balancer.lua:333: in function ‘create_healthchecker’
/usr/local/share/lua/5.1/kong/runloop/balancer.lua:406: in function ‘create_balancer’
/usr/local/share/lua/5.1/kong/runloop/balancer.lua:621: in function ‘on_upstream_event’
/usr/local/share/lua/5.1/kong/runloop/handler.lua:347: in function </usr/local/share/lua/5.1/kong/runloop/handler.lua:342>
[C]: in function ‘xpcall’
/usr/local/share/lua/5.1/resty/worker/events.lua:166: in function ‘do_handlerlist’
/usr/local/share/lua/5.1/resty/worker/events.lua:200: in function ‘do_event_json’
/usr/local/share/lua/5.1/resty/worker/events.lua:342: in function ‘poll’
/usr/local/share/lua/5.1/resty/worker/events.lua:362: in function </usr/local/share/lua/5.1/resty/worker/events.lua:356>’, data={“operation”:“create”,“entity”:{“healthchecks”:{“active”:{“https_verify_certificate”:true,“http_path”:"/ping",“healthy”:{“http_statuses”:[200,302],“interval”:1,“successes”:3},“type”:“http”,“unhealthy”:{“http_statuses”:[429,404,500,501,502,503,504,505],“interval”:1,“timeouts”:3,“http_failures”:3,“tcp_failures”:3},“timeout”:1,“concurrency”:100},“passive”:{“unhealthy”:{“http_failures”:0,“http_statuses”:[429,500,503],“tcp_failures”:0,“timeouts”:0},“healthy”:{“http_statuses”:[200,201,202,203,204,205,206,207,208,226,300,301,302,303,304,305,306,307,308],“successes”:0},“type”:“http”}},“hash_on”:“none”,“id”:“592f0298-52d7-58ff-8bb4-74bc9e32e0c0”,“tags”:{},“name”:“testsant”,“hash_on_cookie_path”:"/",“created_at”:1563431658,“hash_fallback”:“none”,“slots”:3000}}, context: ngx.timer
…
…
2019/07/18 14:34:32 [error] 10960#0: *6145 [lua] events.lua:345: poll(): worker-events: dropping event; waiting for event data timed out, id: 10212, context: ngx.timer
2019/07/18 14:34:32 [error] 10960#0: *6145 [lua] events.lua:345: poll(): worker-events: dropping event; waiting for event data timed out, id: 10213, context: ngx.timer
2019/07/18 14:34:32 [error] 10960#0: *6145 [lua] events.lua:345: poll(): worker-events: dropping event; waiting for event data timed out, id: 10214, context: ngx.timer
2019/07/18 14:34:32 [error] 10960#0: *6145 [lua] events.lua:345: poll(): worker-events: dropping event; waiting for event data timed out, id: 10215, context: ngx.timer
2019/07/18 14:34:32 [error] 10960#0: *6145 [lua] events.lua:345: poll(): worker-events: dropping event; waiting for event data timed out, id: 10216, context: ngx.timer
2019/07/18 14:34:32 [error] 10960#0: *6145 [lua] events.lua:345: poll(): worker-events: dropping event; waiting for event data timed out, id: 10217, context: ngx.timer
2019/07/18 14:34:32 [error] 10972#0: *6142 [lua] events.lua:345: poll(): worker-events: dropping event; waiting for event data timed out, id: 8199, context: ngx.timer
2019/07/18 14:34:32 [error] 10972#0: *6142 [lua] events.lua:345: poll(): worker-events: dropping event; waiting for event data timed out, id: 8200, context: ngx.timer
2019/07/18 14:34:33 [error] 10972#0: *6142 [lua] events.lua:345: poll(): worker-events: dropping event; waiting for event data timed out, id: 8206, context: ngx.timer