# Monitor lua memory

**URL:** <https://discuss.konghq.com/t/monitor-lua-memory/3015>\
**Category:** General\
**Created:** [March 14, 2019, 1:51pm UTC](https://discuss.konghq.com/t/monitor-lua-memory/3015 "2019-03-14T13:51:31Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![likharev](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/likharev/32/414_2.png) [@likharev](https://discuss.konghq.com/u/likharev)\
**Post date:** [March 14, 2019, 1:51pm UTC](https://discuss.konghq.com/t/monitor-lua-memory/3015/1 "2019-03-14T13:51:31Z")

</div>

what is the best way to monitory LUA memory on kong, last week all our kong cluster eventually died spiking CPU, I suspect this is due to memory pressure on LUA heap.

---

<div class="post-metadata">

**Author:** ![flowdopip](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/flowdopip/32/1704_2.png) [@flowdopip](https://discuss.konghq.com/u/flowdopip)\
**Post date:** [March 14, 2019, 5:40pm UTC](https://discuss.konghq.com/t/monitor-lua-memory/3015/2 "2019-03-14T17:40:37Z")

</div>

Hi,

Can you provide more information about the cluster setup (config and size) and plugins installed?

---

<div class="post-metadata">

**Author:** ![likharev](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/likharev/32/414_2.png) [@likharev](https://discuss.konghq.com/u/likharev)\
**Post date:** [March 14, 2019, 6:30pm UTC](https://discuss.konghq.com/t/monitor-lua-memory/3015/3 "2019-03-14T18:30:26Z")

</div>

AWS  
3 nodes, t2.large, 2 CPU, 8 GB, each node in separate availability zone, behind ALB

about 10 APIs, 20 upstreams, 3 targets in each upstream, each API uses upstreams names to proxy.

similar set of plugins, like cors, key-auth, acl, we have also custom plugin based on PDK, that allow fallback to different zone, doing set\_upstream

---

<div class="post-metadata">

**Author:** ![flowdopip](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/flowdopip/32/1704_2.png) [@flowdopip](https://discuss.konghq.com/u/flowdopip)\
**Post date:** [March 15, 2019, 10:34am UTC](https://discuss.konghq.com/t/monitor-lua-memory/3015/4 "2019-03-15T10:34:46Z")

</div>

And about the traffic?  
Do you have logs?  
Do you have metrics?  
What database are you using?  
Are you in debug?

---

<div class="post-metadata">

**Author:** ![likharev](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/likharev/32/414_2.png) [@likharev](https://discuss.konghq.com/u/likharev)\
**Post date:** [March 15, 2019, 3:24pm UTC](https://discuss.konghq.com/t/monitor-lua-memory/3015/5 "2019-03-15T15:24:02Z")

</div>

traffic about 10 req/sec  
I do have logs, I do have logs and I do have metrics, what specifically do you need?  
database is cassandra, multi region  
no I’m not in debug as I’m in production, but I probably can spinn-up additional instance with debug option.

what I don’t like about this situation, is that all 3 nodes when unresponsive with 100% CPU in relatively short time. LUA memory is one of my guesses, so wondering how I can monitor it, without additional plugin and

> collectgarbage(“count”)

> 

to put more context to this.

- in kong log files on all instances ( access, admin\_access, error ) there is a gap( no activity ) in 90 min for the duration of CPU spike.

- any metrics you choose, I have them all, all is good, except CPU utilization for nginx which took ~50% and in 10 min allocated 100%, nothing was frozen on the instance, I was able to ssh, look around, stop and start kong.

---

<div class="post-metadata">

**Author:** ![flowdopip](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/flowdopip/32/1704_2.png) [@flowdopip](https://discuss.konghq.com/u/flowdopip)\
**Post date:** [March 15, 2019, 4:59pm UTC](https://discuss.konghq.com/t/monitor-lua-memory/3015/6 "2019-03-15T16:59:56Z")

</div>

What is the version of Kong.  
I´m asking this things because on past i have exactly the same behaviour but after moving everything to the latest version 1.0.1 and moving to Services and Routes all the system has established.  
But at the same time i switch from container to dedicated VM´s.

---

<div class="post-metadata">

**Author:** ![likharev](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/likharev/32/414_2.png) [@likharev](https://discuss.konghq.com/u/likharev)\
**Post date:** [March 15, 2019, 6:03pm UTC](https://discuss.konghq.com/t/monitor-lua-memory/3015/7 "2019-03-15T18:03:48Z")

</div>

this is kong 0.14.1, we are not ready to upgrade yet, but will probably soon.  
also we do use services/routes and dedicated VM ( ec2 )

---

<div class="post-metadata">

**Author:** ![p0pr0ck5](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/p0pr0ck5/32/987_2.png) [@p0pr0ck5](https://discuss.konghq.com/u/p0pr0ck5)\
**Post date:** [March 20, 2019, 5:05am UTC](https://discuss.konghq.com/t/monitor-lua-memory/3015/8 "2019-03-20T05:05:03Z")

</div>

Just a note, if you’re running on a burstable T2 instance, you need to check to ensure you haven’t burned through all your CPU cycle credits and the node isn’t getting throttled.

---

<div class="post-metadata">

**Author:** ![likharev](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/likharev/32/414_2.png) [@likharev](https://discuss.konghq.com/u/likharev)\
**Post date:** [March 20, 2019, 8:24pm UTC](https://discuss.konghq.com/t/monitor-lua-memory/3015/9 "2019-03-20T20:24:31Z")

</div>

I don’t want to mix CPU into this discussion and specifically talk about LUA memory monitoring. Actually that would be a very nice feature for the kong metric, LUA memory size, free/used

---

<div class="post-metadata">

**Author:** ![p0pr0ck5](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/p0pr0ck5/32/987_2.png) [@p0pr0ck5](https://discuss.konghq.com/u/p0pr0ck5)\
**Post date:** [May 6, 2019, 3:42pm UTC](https://discuss.konghq.com/t/monitor-lua-memory/3015/10 "2019-05-06T15:42:41Z")

</div>

Can you describe what _specific_ metrics you want to monitor within Lua? BTW, I don’t think you’re going to have much luck with it, the Lua VM details are hidden away for obvious reason. You could look at the memory allocation made by each worker process from a system perspective, but things like GC details, etc, probably can’t (shouldn’t) be exposed in the context. If you need that kind of detail for troubleshooting, you might want to have a grok of SystemTap and some accompanying tools: [https://github.com/openresty/stapxx](https://github.com/openresty/stapxx)

---

<div class="post-metadata">

**Author:** ![likharev](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/likharev/32/414_2.png) [@likharev](https://discuss.konghq.com/u/likharev)\
**Post date:** [May 9, 2019, 7:29pm UTC](https://discuss.konghq.com/t/monitor-lua-memory/3015/11 "2019-05-09T19:29:38Z")

</div>

this is what we’ve done, just a note, we already using modified statsd plugin, we need it to send dimensions.

we added:  
logger:send\_statsd(“lua.memory”, collectgarbage(“count”) \* 1024, logger.stat\_types.gauge, nil)

reason, we we trying to identify high CPU usage, and that part of the system is a black box, at least we have some metrics.

looks like this:

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/konghq/original/1X/bb4ee176f5e29194b75617f51f9996682a181adf.png)

---

<div class="post-metadata">

**Author:** ![nitsshukla](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/nitsshukla/32/1335_2.png) [@nitsshukla](https://discuss.konghq.com/u/nitsshukla)\
**Post date:** [September 16, 2019, 5:52am UTC](https://discuss.konghq.com/t/monitor-lua-memory/3015/12 "2019-09-16T05:52:48Z")

</div>

Take a look at metrics exposed by Prometheus specially in this git-repo

> kong\_memory\_lua\_shared\_dict\_total\_bytes  
> kong\_memory\_workers\_lua\_vms\_bytes

> **[GitHub - Kong/kong-plugin-prometheus: Prometheus plugin for Kong - this...](https://github.com/Kong/kong-plugin-prometheus/)**
>
> Prometheus plugin for Kong - this plugin has been moved into https://github.com/Kong/kong, please open issues and PRs in that repo - GitHub - Kong/kong-plugin-prometheus: Prometheus plugin for Kon...

Also, you can use collectgarbage API [http://luatut.com/collectgarbage.html](http://luatut.com/collectgarbage.html)
