Route configuration is deleted for DB-Less on 1.3.0 during high traffic

We recently migrated from 0.14.1 to 1.3.0 and switched to DB-less with the migration from Postgre. After migrating we’ve noticed intermittent 404s with the message { "message": "no Route matched with those values" } during periods with higher than usual traffic, around 5,000 ~ 10,000 requests per minute. We have roughly 200 combined services, routes, and plugins in use. Most of our plugins are custom plugins for header manipulation and authentication against a Redis instance. We never ran into this issue while on 0.14.1.

We’ve confirmed that during this periods the route configuration is removed from the cache and all of our previously configured services and routes are deleted. There’s no notable error messages displayed when the cache is cleared besides worker process 43 exited on signal 11.

We’ve increased the mem_cache_size to 1024m which has not had much effect. Many of the values in our nginx configuration file come from https://github.com/Kong/kong/blob/1.3.0/spec/fixtures/custom_nginx.template. At this point we’re unsure of what the issue could be. We’re thinking there might be some sort of memory leak as the issue tends to resolve itself when we restart the existing services, but still need to do more investigation before we can be sure.

Any suggested workarounds or fixes would be greatly appreciated. Thank you!

Can anyone please take a look at this?

We have increased the memory, tuned nginx worker processes, upgraded kong_cache_size. It is still happening!

We have a watchdog script currently checking to see if the config gets lost but this is not a good long-term solution.

This seems to be another report or a related effect due to the same (unknown) cause of Possible memory leak - Kong 1.4 & KIC 0.6.1.


© 2019 Kong Inc.    Terms  •  Privacy  •  FAQ