DB-less kong-1.3.0, the upstream doesn't work, got 'failed posting event "upstreams" by "balancer"; no memory'

Questions
1

Hi, guys. I got a problem, please following the message, hope helpful.

environment:

  • CentOS release 6.2
  • mem_cache_size = 128m (by default)
2019/10/16 15:25:01 [error] 26259#0: *139856673 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/resty/dns/balancer/base.lua:1206: attempt to index field 'lastQuery' (a nil value)
2019/10/16 15:25:01 [error] 26279#0: *139856704 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/resty/dns/balancer/base.lua:1206: attempt to index field 'lastQuery' (a nil value)
2019/10/16 15:25:01 [error] 26278#0: *139856712 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/resty/dns/balancer/base.lua:1206: attempt to index field 'lastQuery' (a nil value)
2019/10/16 15:25:01 [error] 26260#0: *139855974 [lua] events.lua:273: post(): worker-events: failed posting event "upstreams" by "balancer"; no memory, client: 127.0.0.1, server: kong_admin, request: "POST /config HTTP/1.1", host: "localhost:8001"
2019/10/16 15:25:01 [error] 26260#0: *139855974 [lua] handler.lua:329: failed broadcasting upstream create to workers: failed posting event "upstreams" by "balancer"; no memory, client: 127.0.0.1, server: kong_admin, request: "POST /config HTTP/1.1", host: "localhost:8001"
2019/10/16 15:25:01 [error] 26258#0: *139856000 [lua] events.lua:273: post(): worker-events: failed posting event "upstreams" by "balancer"; no memory, context: ngx.timer
2019/10/16 15:25:01 [error] 26258#0: *139856000 [lua] handler.lua:329: failed broadcasting upstream create to workers: failed posting event "upstreams" by "balancer"; no memory, context: ngx.timer
2019/10/16 15:25:01 [error] 26263#0: *139856023 [lua] events.lua:273: post(): worker-events: failed posting event "upstreams" by "balancer"; no memory, context: ngx.timer
2019/10/16 15:25:01 [error] 26263#0: *139856023 [lua] handler.lua:329: failed broadcasting upstream create to workers: failed posting event "upstreams" by "balancer"; no memory, context: ngx.timer
2019/10/16 15:25:01 [error] 26274#0: *139856882 [lua] events.lua:273: post(): worker-events: failed posting event "upstreams" by "balancer"; no memory, context: ngx.timer
2019/10/16 15:25:01 [error] 26274#0: *139856882 [lua] handler.lua:329: failed broadcasting upstream create to workers: failed posting event "upstreams" by "balancer"; no memory, context: ngx.timer

and addtional steps I did:

  1. dump by kong config db_export kong.yml
  2. post kong.yml by the /config API
  3. every 5s to repeat two steps

btw, I am not sure if it will show again.

2

Hi jayce!

Thanks for your report. It seems that this issue has been already addressed and will be part of the 1.4.0 release, you can check the solution in this pull request: https://github.com/Kong/kong/pull/5014.

If your environment allows using a release candidate version, could you try the 1.4.0rc2 and get back to us with some results?

Thank you,
Vinicius

3

Thanks for your reply. I will try 1.4.0rc2 soon and get back to you.

4

Thanks again, and I have tried the 1.4.0rc2 about 6 hours maybe, it didn’t happen.

1 Like