I had already been working on trying to reproduce and narrow this down in our test environment, and I’m fairly confident that I’ve found which plugin is triggering this.
Disabling the plugin on the particular route that I’m testing keeps the CPU load at a reasonable level, but disabling the entire access handler function body (with a if true then; return; end at the top of the function) does not stop the issue from occurring. This implies that the problem is more “subtle” and is happening within Kong itself.
Edit: I missed disabling part of the plugin’s code. Disabling all of the code in the plugin’s access handler does actually solve the problem. Now I’m testing with code to bail out of the access handler at various points to try to track down the actual problem.
So it seems the problem is around instantiating a class in this particular plugin’s execute() function. I have no idea why this worked fine in Kong 0.13 or why it’s a problem now.
It looks like I finally figured this one out. The problem was a classic pass-by-ref issue and adding an item to a list on every request. The list from the conf object was growing to thousands of entries as Kong took more requests, causing it to take more time to iterate through the list.
Why this happened with the upgrade is unclear. It could be a change in the underlying plugin framework since 0.13, or perhaps due to the switch to the DB-less/declarative config. Either way, it was fixed by doing a “copy” of the table to a new var (rather than a simple assignment) before adding new items to it.