Plugin Cache invalidation problem

Installation/Setup
1

i am currently trying to set up kong 1.0 with 1 cassandra node. both of them are running on docker. in this setup, i added around 72 routes, 72 services, around 10 cnsumers, with 3 or more plugins that is applied when a request comes in (which are applied either in route, or combination of route and consumer). the problem is, when accessing the route, the plugins is not executed at all. I need to restart kong to make the plugins take effect. any idea on generic cassandra setup mistake that might make this happens?

here is my docker configuration

docker run --rm -d \ 
-e "CASSANDRA_BROADCAST_ADDRESS=192.168.111.10" \
-e "CASSANDRA_RACK=RACK1" \
-e "CASSANDRA_SEEDS=192.168.111.10" \
-e "CASSANDRA_DC=DC1" \
-e "CASSANDRA_LISTEN_ADDRESS=192.168.111.10" \
-e "CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch" \
-e "CASSANDRA_CLUSTER_NAME=a-cluster" \
-e "CASSANDRA_RPC_ADDRESS=192.168.111.10" \
--net=host \
cassandra:3.11.3


docker run --rm \
-e "KONG_LOG_LEVEL=debug" \
-e "KONG_TRUSTED_IPS=192.168.111.3/32" \
-e "KONG_PROXY_LISTEN=0.0.0.0:8000, 0.0.0.0:8443 ssl" \
-e "KONG_PROXY_ERROR_LOG=/dev/stderr" \
-e "KONG_PROXY_ACCESS_LOG=/dev/stdout" \
-e "KONG_ADMIN_LISTEN=0.0.0.0:8001, 0.0.0.0:8444 ssl" \
-e "KONG_ADMIN_ACCESS_LOG=/dev/stdout" \
-e "KONG_ADMIN_ERROR_LOG=/dev/stderr" \
-e "KONG_NGINX_USER=nobody root" \
-e "KONG_DATABASE=cassandra" \
-e "KONG_DB_UPDATE_PROPAGATION=15" \
-e "KONG_CASSANDRA_CONTACT_POINTS=192.168.111.10" \
-e "KONG_CASSANDRA_DATA_CENTERS=DC1:1" \
-e "KONG_CASSANDRA_LOCAL_DATACENTER=DC1" \
-e "KONG_CASSANDRA_LB_POLICY=DCAwareRoundRobin" \
-e "KONG_CASSANDRA_REPL_STRATEGY=NetworkTopologyStrategy" \
-e "KONG_CASSANDRA_CONSISTENCY=LOCAL_QUORUM" \
-e "KONG_LUA_SSL_TRUSTED_CERTIFICATE=/etc/ssl/certs/ca-certificates.crt" \
-e "KONG_LUA_SSL_VERIFY_DEPTH=2 \
-v /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt \
--net=host \
kong:1.0.0-alpine kong migrations bootstrap

docker run --rm -d \
-e "KONG_LOG_LEVEL=debug" \
-e "KONG_TRUSTED_IPS=192.168.111.3/32" \
-e "KONG_PROXY_LISTEN=0.0.0.0:8000, 0.0.0.0:8443 ssl" \
-e "KONG_PROXY_ERROR_LOG=/dev/stderr" \
-e "KONG_PROXY_ACCESS_LOG=/dev/stdout" \
-e "KONG_ADMIN_LISTEN=0.0.0.0:8001, 0.0.0.0:8444 ssl" \
-e "KONG_ADMIN_ACCESS_LOG=/dev/stdout" \
-e "KONG_ADMIN_ERROR_LOG=/dev/stderr" \
-e "KONG_NGINX_USER=nobody root" \
-e "KONG_DATABASE=cassandra" \
-e "KONG_DB_UPDATE_PROPAGATION=15" \
-e "KONG_CASSANDRA_CONTACT_POINTS=192.168.111.10" \
-e "KONG_CASSANDRA_DATA_CENTERS=DC1:1" \
-e "KONG_CASSANDRA_LOCAL_DATACENTER=DC1" \
-e "KONG_CASSANDRA_LB_POLICY=DCAwareRoundRobin" \
-e "KONG_CASSANDRA_REPL_STRATEGY=NetworkTopologyStrategy" \
-e "KONG_CASSANDRA_CONSISTENCY=LOCAL_QUORUM" \
-e "KONG_LUA_SSL_TRUSTED_CERTIFICATE=/etc/ssl/certs/ca-certificates.crt" \
-e "KONG_LUA_SSL_VERIFY_DEPTH=2 \
--net=host \
-v /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt \
kong:1.0.0-alpine
2

i’ve tried it also with postgres but same result.

tried with this docker :

docker run --rm -d \ 
-e "POSTGRES_DB=kong" \
-e        "POSTGRES_USER=kong" \
--net=host \
postgres:10.1-alpine

the kong container is the same with the above, just with a modified environment

KONG_DATABASE="postgres"
KONG_PG_HOST="192.168.111.10"
KONG_DB_UPDATE_PROPAGATION="0"
3

Interesting, I am using C* on 1.0 too in a sandbox but have yet to see this behavior, I originally spotted this:

Which lots of plugin entries would cause a major delay on the first cold_cache route invoke due to some db checks that did not need to be executed in a hot path like it was. But yet to produce your issue. I am curious if when 1.0.1 comes out in a bit if this behavior continues to be seen. Could try manually patching the version you run with all those perf changes referencing that github issue and see if you still see the problem?

4

i’ve created an issue and a repo to reproduce it.

Could try manually patching the version you run with all those perf changes referencing that github issue and see if you still see the problem?

i’ll give it a try. Hope that fixes the issue. thank you very much for the refference

1 Like
5

tried with this commit merged to master but still no luck

6

And this one: https://github.com/Kong/kong/pull/4179 ? Otherwise may have make a kong github issue and see if they can reproduce.

7

That was merged to master 5 days ago right? If that’s the case then yes that one too