Worker-events errors after ingress-controller upgrade

Questions
1

We recently upgraded our ingress-controller from 0.3.0 to 0.9.1 and started to see the following errors in our logs since:

2020/09/02 15:50:55 [error] 23#0: *20834880 [lua] events.lua:273: post(): worker-events: failed posting event “upstreams” by “balancer”; no memory, client: 127.0.0.1, server: kong_admin, request: “PUT /upstreams/6eca17df-94e1-421d-8c25-97368f44b60d HTTP/1.1”, host: “127.0.0.1:8444”
2020/09/02 15:50:57 [error] 23#0: *20833853 [lua] events.lua:364: post(): worker-events: dropping event; waiting for event data timed out, id: 766726, client: 127.0.0.1, server: kong_admin, request: “PUT /upstreams/0519f25a-7744-46c2-ad42-b815f29dd3c5 HTTP/1.1”, host: “127.0.0.1:8444”

Kong Version: 2.0.4
Ingress Controller: 0.9.1
PostgreSQL: 9.6
Kubernetes: 1.14.10 GKE

I have found a few topics related to this error but they seem to be related to db-less (which we’re not using) and was supposedly solved in Kong 1.4.0.

I have tried the following suggestions I’ve seen elsewhere without any luck:

  • increasing memory available to the pods
  • setting KONG_NGINX_WORKER_PROCESSES to 1
  • bumping KONG_MEM_CACHE_SIZE to 256m

Here is the information from the /status endpoint:

{
“database”: {
“reachable”: true
},
“memory”: {
“workers_lua_vms”: [
{
“http_allocated_gc”: “0.04 MiB”,
“pid”: 23
}
],
“lua_shared_dicts”: {
“kong_locks”: {
“allocated_slabs”: “0.06 MiB”,
“capacity”: “8.00 MiB”
},
“kong”: {
“allocated_slabs”: “0.04 MiB”,
“capacity”: “5.00 MiB”
},
“kong_process_events”: {
“allocated_slabs”: “5.00 MiB”,
“capacity”: “5.00 MiB”
},
“kong_db_cache_miss”: {
“allocated_slabs”: “0.08 MiB”,
“capacity”: “12.00 MiB”
},
“kong_healthchecks”: {
“allocated_slabs”: “0.26 MiB”,
“capacity”: “5.00 MiB”
},
“kong_cluster_events”: {
“allocated_slabs”: “0.10 MiB”,
“capacity”: “5.00 MiB”
},
“kong_core_db_cache_miss”: {
“allocated_slabs”: “0.08 MiB”,
“capacity”: “12.00 MiB”
},
“kong_core_db_cache”: {
“allocated_slabs”: “2.74 MiB”,
“capacity”: “256.00 MiB”
},
“prometheus_metrics”: {
“allocated_slabs”: “0.04 MiB”,
“capacity”: “5.00 MiB”
},
“kong_db_cache”: {
“allocated_slabs”: “1.50 MiB”,
“capacity”: “256.00 MiB”
},
“kong_rate_limiting_counters”: {
“allocated_slabs”: “0.08 MiB”,
“capacity”: “12.00 MiB”
}
}
},
“server”: {
“connections_writing”: 1,
“total_requests”: 202326,
“connections_handled”: 47334,
“connections_accepted”: 47334,
“connections_reading”: 0,
“connections_active”: 2,
“connections_waiting”: 1
}
}

Any advice or other suggestions would be very much appreciated, thank you!