# Custom Lua plugin worker failing to flush buffered telemetry payloads during async HTTP dispatch **URL:** \ **Category:** Questions\ **Created:** [July 22, 2026, 4:09pm UTC](https://discuss.konghq.com/t/custom-lua-plugin-worker-failing-to-flush-buffered-telemetry-payloads-during-async-http-dispatch/14063 "2026-07-22T16:09:41Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![lara887](https://avatars.discourse-cdn.com/v4/letter/l/a9a28c/32.png) [@lara887](https://discuss.konghq.com/u/lara887)\ **Post date:** [July 22, 2026, 4:09pm UTC](https://discuss.konghq.com/t/custom-lua-plugin-worker-failing-to-flush-buffered-telemetry-payloads-during-async-http-dispatch/14063/1 "2026-07-22T16:09:42Z")
Hey everyone, I am currently running into an issue on Kong Gateway where a custom Lua plugin is failing to reliably flush buffered request telemetry payloads to an external collector under high concurrent load, and I am trying to figure out if this is related to how the nginx worker thread lifecycle handles asynchronous sockets. For some context on our architecture, our team uses Kong Gateway to route and monitor incoming API traffic from an external game state synchronization server. Our backend services process real-time events and user automation data from sandboxed testing environments. We manage a fleet of QA instances where developers run performance scripts through a roblox executor setup to test server-side anti-cheat logic and bandwidth limits under heavy traffic. We keep our core developer guidelines and public documentation organized on an open [free](https://deltaexector.com/) index workspace, and we pass these event metrics through our Kong API Gateway layers to authenticate and rate-limit client requests. The primary issue we are facing is that during peak traffic bursts, our custom logging phase plugin begins dropping background log batches. We are using timer wrappers to spawn background tasks and dispatch JSON payload buffers via TCP sockets, but when request rates spike, Kong starts throwing errors indicating that there are too many pending timers in the error log. The worker processes seem to hit their timer limit, causing pending telemetry batches to get discarded before they ever reach the network layer. In addition to the dropped telemetry timers, we are noticing a secondary memory buildup on the Gateway nodes. When these timer queues back up under load, RAM usage across the worker processes climbs steadily and fails to release back to baseline levels even after client traffic settles down, eventually forcing us to reload the Kong process to reclaim memory. Has anyone else encountered timer queue exhaustion when using custom Lua plugins to stream high-volume request metrics through Kong Gateway, and what is the recommended buffer configuration or batching pattern to handle high-concurrency log flushing without overloading the worker event loop? ---
**Author:** ![discourse\_ai\_spam](https://avatars.discourse-cdn.com/v4/letter/d/c68b51/32.png) [@discourse\_ai\_spam](https://discuss.konghq.com/u/discourse_ai_spam)\ **Post date:** [July 22, 2026, 4:09pm UTC](https://discuss.konghq.com/t/custom-lua-plugin-worker-failing-to-flush-buffered-telemetry-payloads-during-async-http-dispatch/14063/2 "2026-07-22T16:09:45Z")
---
**Author:** ![system](https://canada1.discourse-cdn.com/flex036/uploads/konghq/original/2X/8/8651049fb84484276f21c943c975876b17c63898.png) [@system](https://discuss.konghq.com/u/system)\ **Post date:** [July 24, 2026, 8:01pm UTC](https://discuss.konghq.com/t/custom-lua-plugin-worker-failing-to-flush-buffered-telemetry-payloads-during-async-http-dispatch/14063/3 "2026-07-24T20:01:28Z")
---
**Author:** ![rick](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/rick/32/4446_2.png) [@rick](https://discuss.konghq.com/u/rick)\ **Post date:** [July 27, 2026, 1:20pm UTC](https://discuss.konghq.com/t/custom-lua-plugin-worker-failing-to-flush-buffered-telemetry-payloads-during-async-http-dispatch/14063/4 "2026-07-27T13:20:48Z")
@lara887 I passed your question through our documentation agent, please see the repsonse for information regarding your issue. * * * ## Timer Exhaustion & Batch Queue Configuration in Kong Gateway This is a well-known issue with custom Lua plugins that spawn individual timers per request for background log dispatch. Kong’s built-in plugins have addressed exactly this problem, and the recommended pattern is to use **internal batch queues** instead of per-request timers. ### Root Cause When a custom plugin spawns a new timer for every request during the log phase, timer counts grow proportionally with traffic. Under high concurrency, this exhausts the worker’s available timer slots, producing the `too many pending timers` error and causing batches to be silently dropped. [[Gateway 2.8.0.0](https://developer.konghq.com/gateway/changelog/#2-8-0-0-bugfix)] This is precisely why Kong added batch queues to the Datadog and StatsD plugins — to fix the same `lua_max_running_timers are not enough` error caused by creating a new timer per request during the log phase. [[Gateway 2.8.2.2](https://developer.konghq.com/gateway/changelog/#2-8-2-2)] * * * ### Recommended Pattern: Use a Single Queue with Batching Instead of spawning a timer per request, the correct pattern is to enqueue entries and use **a single background timer** to drain the queue in batches. This is how all of Kong’s built-in logging/telemetry plugins (HTTP Log, OpenTelemetry, Datadog, StatsD, Zipkin) work. [[Plugin Queues](https://developer.konghq.com/gateway/resource-sizing-guidelines/#plugin-queues)] Key queue parameters you should configure for your custom plugin (or migrate to a built-in plugin if applicable): | Parameter | Purpose | |—|—| | `queue.max_batch_size` | Max entries sent per upstream request (reduces upstream concurrency) | | `queue.max_entries` | Max entries waiting in queue (default: 10,000) | | `queue.max_bytes` | Max bytes waiting in queue | | `queue.concurrency_limit` | Number of background delivery timers (default: 1) | | `queue.initial_retry_delay` | Wait before first retry on failure | | `queue.max_retry_delay` | Cap on exponential backoff | | `queue.max_retry_time` | Total time before giving up on a failing batch | [[HTTP Log Queuing](https://developer.konghq.com/plugins/http-log/#queuing); [OpenTelemetry Queuing](https://developer.konghq.com/plugins/opentelemetry/#queuing)] * * * ### Key Behaviors to Be Aware Of - **Queues are per-worker and in-memory only.** They are not shared between Nginx worker processes. For capacity planning, multiply your queue limits by the number of workers. [[HTTP Log Queuing](https://developer.konghq.com/plugins/http-log/#queuing)] - **Memory buildup** is expected when the upstream collector is unavailable and the queue fills up. The queue will warn at 80% capacity and begin dropping the **oldest** entries once `max_entries` is reached. This is the designed behavior to bound memory usage. [[Plugin Queues](https://developer.konghq.com/gateway/resource-sizing-guidelines/#plugin-queues)] - **On shutdown** , queues are flushed automatically, so in-flight batches are not lost during graceful restarts. [[OpenTelemetry Queuing](https://developer.konghq.com/plugins/opentelemetry/#queuing)] - **Queued content is not durable** — power loss or unexpected worker crashes will lose buffered data. [[HTTP Log Queuing](https://developer.konghq.com/plugins/http-log/#queuing)] * * * ### Practical Recommendations 1. **Refactor your custom plugin** to use a single shared queue with one background timer, rather than spawning a timer per log entry. 2. **Tune `queue.max_batch_size`** — Kong increased the default for OpenTelemetry to 200 in 3.6.1.0 as a performance improvement. [[Gateway 3.6.1.0](https://developer.konghq.com/gateway/changelog/#3-6-1-0)] 3. **Size `queue.max_entries`** based on your expected burst duration × request rate per worker. The default of 10,000 is a reasonable starting point. [[Plugin Queues](https://developer.konghq.com/gateway/resource-sizing-guidelines/#plugin-queues)] 4. **Test under load with the upstream collector intentionally unavailable** to observe peak memory consumption before setting limits for production. [[Plugin Queues](https://developer.konghq.com/gateway/resource-sizing-guidelines/#plugin-queues)] If migrating to a built-in plugin is feasible (e.g., HTTP Log for JSON payloads over HTTP/HTTPS), that would be the most straightforward path since the queuing infrastructure is already implemented and battle-tested. ---
**Author:** ![rick](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.konghq.com/rick/32/4446_2.png) [@rick](https://discuss.konghq.com/u/rick)\ **Post date:** [July 27, 2026, 1:21pm UTC](https://discuss.konghq.com/t/custom-lua-plugin-worker-failing-to-flush-buffered-telemetry-payloads-during-async-http-dispatch/14063/5 "2026-07-27T13:21:34Z")
See [https://developer.konghq.com/](https://developer.konghq.com/) and the AI Agent help available at the top. HOpe this helps