Kong 1.4.0 - [lua] prometheus.lua:360: log_error(): Error while setting 'no memory'

Questions
1

Hello,

We are running Kong 1.4.0 on linux. We are using the Prometheus plugin and are getting a ton of errors (has 3,099,482 in a day) similar to this below in the /usr/local/kong/logs/error.log file.

2019/12/10 21:41:10 [error] 13010#0: *8935225 [lua] prometheus.lua:360: log_error(): Error while setting 'latency_bucket{type="kong",service="my-cool-service",route="eb443bee-eea7-42e7-8512-91a7f8fa1030",le="60000.0"}' to '1': 'no memory' while logging request, client: 70.132.33.71, server: kong, request: "GET /api/my-cool-service/v2/historical HTTP/1.1", upstream: "http://10.252.235.119:80/v2/historical", host: "my-cool-domain.com"

When we run curl http://localhost:8001/status it shows Prometheus metrics of this below.

      "prometheus_metrics": {
    "allocated_slabs": "5.00 MiB",
    "capacity": "5.00 MiB"
  },

We found https://github.com/Kong/kong/issues/4589 that recommends to set nginx_http_lua_shared_dict=prometheus metrics 10m in /etc/kong/kong.conf file and restarting kong by running service kong stop and service kong start. That failed with the error below.

Error: could not prepare Kong prefix at /usr/local/kong: nginx configuration is invalid (exit code 1):

nginx: [emerg] invalid number of arguments in “lua_shared_dict” directive in /usr/local/kong/nginx-kong.conf:30
nginx: configuration file /usr/local/kong/nginx.conf test failed

Run with --v (verbose) or --vv (debug) for more details
Error: could not prepare Kong prefix at /usr/local/kong: nginx configuration is invalid (exit code 1):
nginx: [emerg] invalid number of arguments in “lua_shared_dict” directive in /usr/local/kong/nginx- kong.conf:30
nginx: configuration file /usr/local/kong/nginx.conf test failed

So we tried setting it to lua_shared_dict=prometheus metrics 10m and restarted kong. No error from kong restart, but that doesn’t update the prometheus metrics from the status and the errors in the log keep showing up.

Any ideas how to fix the no memory errors in the log for prometheus? We think increasing the lua_shared_dict limit will solve this issue. Our kong services are working just fine. Please advise.

Thank you,
Mike

2

You need nginx_http_lua_shared_dict=prometheus_metrics 10m.

Not the _ before metrics.

3

Hello @hbagdi,

That resolved my issue.

Thank you