Hi all,
I’m running into a couple of issues when using Kong 1.3.0-rc1 databaseless with the Kubernetes Ingress controller.
#1: This health check issue seems to be present all the time, though it gets worse as more routes are deployed.
2019/08/09 18:34:12 [error] 160#0: *147587 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/runloop/balancer.lua:249: attempt to index local 'healthchecker' (a nil value)
stack traceback:
coroutine 0:
/usr/local/share/lua/5.1/kong/runloop/balancer.lua: in function 'callback'
/usr/local/share/lua/5.1/resty/dns/balancer/base.lua:1270: in function </usr/local/share/lua/5.1/resty/dns/balancer/base.lua:1269>, context: ngx.timer
2019/08/09 18:34:42 [crit] 160#0: *101490 [lua] balancer.lua:640: on_upstream_event(): failed creating balancer for cloudproxy.arnavdatabus.svc: timeout waiting for balancer for 00aca459-9400-5dc9-905d-1908b6b36e34, context: ngx.timer
2019/08/09 18:34:42 [error] 160#0: *101490 [lua] balancer.lua:609: on_target_event(): target create: balancer not found for cloudproxy.arnavdatabus.svc, context: ngx.timer
2019/08/09 18:34:42 [error] 160#0: *101490 [lua] events.lua:194: do_handlerlist(): worker-events: event callback failed; source=lua-resty-healthcheck [service.namespace.svc], event=clear, pid=164 error='/usr/local/share/lua/5.1/resty/healthcheck.lua:225: attempt to index field 'targets' (a nil value)
stack traceback:
/usr/local/share/lua/5.1/resty/healthcheck.lua:225: in function 'get_target'
/usr/local/share/lua/5.1/resty/healthcheck.lua:942: in function </usr/local/share/lua/5.1/resty/healthcheck.lua:940>
[C]: in function 'xpcall'
/usr/local/share/lua/5.1/resty/worker/events.lua:185: in function 'do_handlerlist'
/usr/local/share/lua/5.1/resty/worker/events.lua:217: in function 'do_event_json'
/usr/local/share/lua/5.1/resty/worker/events.lua:361: in function 'poll'
/usr/local/share/lua/5.1/resty/healthcheck.lua:1326: in function 'new'
/usr/local/share/lua/5.1/kong/runloop/balancer.lua:339: in function 'create_healthchecker'
/usr/local/share/lua/5.1/kong/runloop/balancer.lua:423: in function 'create_balancer'
/usr/local/share/lua/5.1/kong/runloop/balancer.lua:760: in function 'init'
/usr/local/share/lua/5.1/kong/runloop/handler.lua:812: in function </usr/local/share/lua/5.1/kong/runloop/handler.lua:811>', data={}, context: ngx.timer
#2: The official prometheus plugin does not appear to be working. When querying the endpoint, I only receive 500 internal server errors.
/plugins endpoint
{
"created_at": 1565376474,
"config": {},
"id": "86e6f637-64e2-5672-920c-32e17e76e0fe",
"service": null,
"enabled": true,
"tags": null,
"consumer": null,
"run_on": "first",
"name": "prometheus",
"route": null,
"protocols": [
"grpc",
"grpcs",
"http",
"https"
]
},
/metrics endpoint:
{
"message": "An unexpected error occurred"
}
Kong/logs/error.log
2019/08/09 18:51:07 [error] 162#0: *174040 [kong] exporter.lua:88 prometheus: plugin is not initialized, please make sure 'prometheus_metrics' shared dict is present in nginx template, client: 10.0.85.66, server: kong_prometheus_exporter, request: "GET /metrics HTTP/1.1", host: "10.0.48.81:9542"
2019/08/09 18:51:07 [error] 162#0: *174040 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/pdk/private/phases.lua:66: no phase in kong.ctx.core.phase
stack traceback:
coroutine 0:
[C]: in function 'error'
/usr/local/share/lua/5.1/kong/pdk/private/phases.lua:66: in function 'check_phase'
/usr/local/share/lua/5.1/kong/pdk/response.lua:662: in function 'collect'
content_by_lua(servers.conf:12):3: in main chunk, client: ..., server: kong_prometheus_exporter, request: "GET /metrics HTTP/1.1", host: "...:9542"
#3: Kong fails to create/find upstreams for upstreams. I think this is related to #1, but I’m listing it separately just in case.
kong/logs/error.log
2019/08/09 18:50:44 [error] 193#0: *173385 [lua] balancer.lua:634: on_upstream_event(): upstream not found for 00aca459-9400-5dc9-905d-1908b6b36e34, context: ngx.timer
2019/08/09 18:50:44 [error] 193#0: *173385 [lua] balancer.lua:634: on_upstream_event(): upstream not found for 26c43ca3-80ff-5862-98d5-e63d327ff612, context: ngx.timer
2019/08/09 18:50:44 [error] 193#0: *173385 [lua] balancer.lua:634: on_upstream_event(): upstream not found for be153462-f6dd-5574-8644-80a980f68141, context: ngx.timer
2019/08/09 18:50:44 [error] 193#0: *173385 [lua] balancer.lua:603: on_target_event(): target create: upstream not found for be153462-f6dd-5574-8644-80a980f68141, context: ngx.timer
2019/08/09 18:50:44 [error] 193#0: *173385 [lua] balancer.lua:603: on_target_event(): target create: upstream not found for 26c43ca3-80ff-5862-98d5-e63d327ff612, context: ngx.timer
2019/08/09 18:50:44 [error] 193#0: *173385 [lua] balancer.lua:603: on_target_event(): target create: upstream not found for 00aca459-9400-5dc9-905d-1908b6b36e34, context: ngx.timer
In addition I have noticed that after some time, Kong ‘looses’ all of its configuration data. (making me think that Kong itself has crashed). In these cases I notice errors like this:
kong/logs/error.log
019/08/09 18:50:45 [notice] 1#0: signal 17 (SIGCHLD) received from 161
2019/08/09 18:50:45 [alert] 1#0: worker process 161 exited on signal 9
2019/08/09 18:50:45 [notice] 1#0: start worker process 265
2019/08/09 18:50:45 [notice] 1#0: signal 29 (SIGIO) received
/upstreams
{
"next": null,
"data": []
}
/routes
{
"next": null,
"data": []
}
/plugins
{
"next": null,
"data": []
}
Any help would be appreciated,
Thanks,
Arnav