【prometheus】 lua coroutine: memory allocation error: not enough memory

Questions
1

kong version: 0.14.1
prometheus_metrics 2048m

ERROR:

2019/07/19 13:15:41 [error] 19173#0: *33456259895 lua coroutine: memory allocation error: not enough memory
stack traceback:
coroutine 0:
[C]: in function ‘get_keys’
…cal/share/lua/5.1/kong/plugins/prometheus/prometheus.lua:503: in function ‘collect’
…local/share/lua/5.1/kong/plugins/prometheus/exporter.lua:162: in function ‘collect’
/usr/local/share/lua/5.1/kong/plugins/prometheus/api.lua:7: in function </usr/local/share/lua/5.1/kong/plugins/prometheus/api.lua:6>
coroutine 1:
[C]: in function ‘resume’
/usr/local/share/lua/5.1/lapis/application.lua:393: in function ‘handler’
/usr/local/share/lua/5.1/lapis/application.lua:130: in function ‘resolve’
/usr/local/share/lua/5.1/lapis/application.lua:161: in function </usr/local/share/lua/5.1/lapis/application.lua:159>
[C]: in function ‘xpcall’
/usr/local/share/lua/5.1/lapis/application.lua:159: in function ‘dispatch’
/usr/local/share/lua/5.1/lapis/nginx.lua:215: in function ‘serve_admin_api’
content_by_lua(nginx-kong.conf:239):2: in function <content_by_lua(nginx-kong.conf:239):1>, client: 10.23.80.11, server: kong_admin, request: “GET /metrics HTTP/1.1”, host: “10.23.11.22:7001”
2019/07/19 13:15:41 [error] 19173#0: *33456259895 [lua] init.lua:133: handle_error(): /usr/local/share/lua/5.1/lapis/application.lua:397: not enough memory
stack traceback:
[C]: in function ‘get_keys’
…cal/share/lua/5.1/kong/plugins/prometheus/prometheus.lua:503: in function ‘collect’
…local/share/lua/5.1/kong/plugins/prometheus/exporter.lua:162: in function ‘collect’
/usr/local/share/lua/5.1/kong/plugins/prometheus/api.lua:7: in function </usr/local/share/lua/5.1/kong/plugins/prometheus/api.lua:6>

stack traceback:
[C]: in function ‘error’
/usr/local/share/lua/5.1/lapis/application.lua:397: in function ‘handler’
/usr/local/share/lua/5.1/lapis/application.lua:130: in function ‘resolve’
/usr/local/share/lua/5.1/lapis/application.lua:161: in function </usr/local/share/lua/5.1/lapis/application.lua:159>
[C]: in function ‘xpcall’
/usr/local/share/lua/5.1/lapis/application.lua:159: in function ‘dispatch’
/usr/local/share/lua/5.1/lapis/nginx.lua:215: in function ‘serve_admin_api’
content_by_lua(nginx-kong.conf:239):2: in function <content_by_lua(nginx-kong.conf:239):1>, client: 10.23.80.11, server: kong_admin, request: “GET /metrics HTTP/1.1”, host: “10.23.11.22:7001”

2

@thibaultcha HELP…

3

LuaJIT-2.1.0-beta3 - 64x，2G Memory limit ???

Enable GC64 mode for x64.

XCFLAGS+= -DLUAJIT_ENABLE_GC64

@thibaultcha

4

@taotao Hi, buddy

comment from the plugin source code:
– This module uses a single dictionary shared between Nginx workers to keep
– all metrics. Each counter is stored as a separate entry in that dictionary,
– which allows us to increment them using built-in incr method.

and init step from conf_loader.lua:

  if conf.loaded_plugins["prometheus"] then
    local http_directives = conf["nginx_http_directives"]
    local found = false

    for _, directive in pairs(http_directives) do
      if directive.name == "lua_shared_dict"
         and string.find(directive.value, "prometheus_metrics", nil, true)
      then
         found = true
         break
      end
    end

    if not found then
      table.insert(http_directives, {
        name  = "lua_shared_dict",
        value = "prometheus_metrics 5m",
      })
    end
  end

which means the default size of prometheus_metrics dict is 5m.You can try to set a bigger value in /etc/kong/kong.conf, for instance

nginx_http_lua_shared_dict = prometheus_metrics 100m

5

you can use http :8001/status to check if the dict is used up.

if so, change the default size. kong reload and use http :8001/status to check if it takes effect.

some thing like

"prometheus_metrics":{"allocated_slabs":"0.85 MiB","capacity":"100.00 MiB"}
6
  • I use the prometheus plug-in in kong(0.1.4.1) without this API interface