In my current setup, I’ve deployed my Kong server as a container in an AWS ECS service, running behind an ALB. For observability, I’ve enabled the Prometheus plugin to expose Kong metrics and deployed a Prometheus server on an EC2 instance to scrape these metrics. Grafana is used as the dashboard.
According to Prometheus metrics, my Kong instance is receiving around 56 requests per second (RPS). However, when I check the HTTP logs, ALB logs, and ECS logs, there is no trace of these incoming requests, which is confusing. We plan to create alerts for Kong metrics with threshold values using Alertmanager. However, the number of requests logged by Kong and those reported by Prometheus are very different. I need a reliable way to set an alarm if Kong receives more than 1000 requests per minute. Can anyone help with this issue?