【performance】Apply prometheus plugins in Kong Route, which adds an average of 6 milliseconds of response time per request

Questions
1

Apply [prometheus plugins] in Kong Route, which adds an average of 6 milliseconds of response time per request，Can decrease response time?

ver : 0.14.1

2

Hi,

Is your objective improving the subjective user experience or the server load? Let me explain.

The bulk of the work done by the Prometheus plugin happens primarily on the log phase. This phase happens after the response headers and body have already been sent to the API consumer. This means that even if you managed to decrease the time you are mentioning, the effective user experience would remain the same as it is now.

On the other hand, if your server was struggling under the load, then trying to decrease the CPU time spent on the log phase might be one possible way to alleviate the pressure.

Other than attempting to rewrite the plugin code to make it more efficient somehow, here are some alternatives that might help:

  • Increasing the size of the prometheus_metrics shared dictionary
  • Increasing the number of workers / CPUs / memory available for Kong
  • Setting several Kong instances in a cluster, if you have not done that already.
3

Thank you，my cluster CPU and memory utilization very sufficient.

Increasing the size of the prometheus_metrics shared dictionary

Is this file?

kong\conf_loader.lua line:690 ?

if not found then
table.insert(http_directives, {
name = “lua_shared_dict”,
value = “prometheus_metrics 5m”, // here ? change 5m to 200m ??
})
end

or

nginx_kong.lua lua_shared_dict 200m?

4

Prometheus plugin has performance issues
An important problem in the test, apply prometheus plugin CPU utilization 90% , remove the plugin CPU utilization 60%, Are there cpu 30% difference betwee these two.

Is it possible to adjust batch or asynchronous metric?

@ thibaultcha @subnetmarco

5

Apologies for not explaining this clearly. Although modifying Kong’s source code that way is always an option, the recommended way of changing this setting is by using a custom nginx configuration. You will have to add the line:

lua_shared_dict prometheus_metrics 200m;

To your kong/templates/nginx_kong.lua file.

As far as I know there is no way to adjust this “out of the box”, the plugin code doesn’t seem to have any settings that can be tweaked. I will ping the plugin author about this.