There is no route invalidation occurring.

#1

Hello Dears,
Could anyone help me? There is no route invalidation occurring.
The DB_UPDATE_FREQUENCY is set to 5 seconds (default), I add a route on node A, the route works on node A, but on node B e C the route does not work.
I do GET curl http://kong:8001/routes on node B and the route added route_new appears, but when I do curl -k https://kong:8443/route_new the route does not work:
{“message”: “no route matched with those values”}.
I enabled the debug and the event below does not occur:
[lua] cluster_events.lua:248: [cluster_events] new event (channel: ‘invalidations’)
After manual kong reload on node B and C, the route works.
My scenery:
3 Kong CE v1.2.1 running on Docker connected to 1 Postgres 9.6
1 Konga v0.14.3 running on Docker.
Host 1: 1 Kong A, 1 Konga e 1 Postgres.
Host 2: 1 Kong B
Host 3: 1 Kong C
Communication between the Kongs and Postgres is working.

#2

Can you show a response from curl http://kong:8001/routes ?

#3

Might be better to keep your comments in the github issue: https://github.com/Kong/kong/issues/4764 , otherwise we are duplicating threads and eyes.

#4

Hello Nareate,

I collected POST and GET in the 3 Kongs.

The “new” route POST was run on server 192.168.10.3, where it worked without kong reload the GET https://frontkong.myhost.com:8443/new.
On servers 192.168.11.2 and 192.168.12.2 GET https://frontkong.myhost.com:8443/new ran just after kong relaod.

Here are the details:
curl -v …8001/routes

  • Trying 192.168.10.3:8001…
  • TCP_NODELAY set
  • Connected to kong (192.168.10.3) port 8001 (#0)

GET /routes HTTP/1.1
Host: kong:8001
User-Agent: curl/7.65.1
Accept: /

  • Mark bundle as not supporting multiuse
    < HTTP/1.1 200 OK
    < Date: Wed, 03 Jul 2019 20:31:27 GMT
    < Content-Type: application/json; charset=utf-8
    < Connection: keep-alive
    < Access-Control-Allow-Origin: *
    < Server: kong/1.2.1
    < Content-Length: 434
    <
  • Connection #0 to host kong left intact
    {“next”:null,“data”:[{“id”:“e5e665cd-660d-4be7-88e4-db0dcaba8512”,“tags”:null,“paths”:["/new"],“destinations”:null,“protocols”:[“https”],“created_at”:1562196258,“snis”:null,“hosts”:[“frontkong-myhost-com-br”],“name”:“RT-NEW”,“preserve_host”:false,“regex_priority”:0,“strip_path”:true,“sources”:null,“updated_at”:1562196258,“https_redirect_status_code”:426,“service”:{“id”:“e247d460-0e17-456a-a23c-0f5d972b0700”},“methods”:[“GET”]}]}

curl -vk …8443/new

  • Trying 192.168.10.3:8443…
  • TCP_NODELAY set
  • Connected to frontkong-myhost-com-br (192.168.10.3) port 8443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: none
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: CN=*.myhost.com.br
  • start date: May 9 00:00:00 2018 GMT
  • expire date: Aug 11 00:00:00 2020 GMT
  • issuer: C=US; O=DigiCert Inc; OU=www-digicert-com; CN=RapidSSL RSA CA 2018
  • SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x55771bf91460)

GET /new HTTP/2
Host: frontkong-myhost-com-br:8443
User-Agent: curl/7.65.1
Accept: /

  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • old SSL session ID is stale, removing
  • Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
    < HTTP/2 200
    < content-type: text/html;charset=utf-8
    < date: Wed, 03 Jul 2019 20:17:50 GMT
    < server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
    < x-frame-options: deny
    < x-xss-protection: 1
    < x-content-type-options: nosniff
    < content-security-policy: default-src ‘none’ ; script-src ‘self’ ‘unsafe-inline’ ‘unsafe-eval’ ; style-src ‘self’ ‘unsafe-inline’ ; img-src ‘self’ …media-rundeck-org ; font-src ‘self’ data: ; connect-src ‘self’ ; form-action ‘self’ ;
    < x-application-context: application:production:4440
    < expires: Thu, 01 Jan 1970 00:00:00 GMT
    < content-language: en-US
    < set-cookie: JSESSIONID=node022e2njuu30lc1y1yxdmjsg1zt396.node0;Path=/;Secure;HttpOnly
    < x-kong-upstream-latency: 22
    < x-kong-proxy-latency: 5
    < via: kong/1.2.1
    <

##################################################################################################################

curl -v …8001/routes

  • Trying 192.168.11.2:8001…
  • TCP_NODELAY set
  • Connected to kong (192.168.11.2) port 8001 (#0)

GET /routes HTTP/1.1
Host: kong:8001
User-Agent: curl/7.65.1
Accept: /

  • Mark bundle as not supporting multiuse
    < HTTP/1.1 200 OK
    < Date: Wed, 03 Jul 2019 20:28:10 GMT
    < Content-Type: application/json; charset=utf-8
    < Connection: keep-alive
    < Access-Control-Allow-Origin: *
    < Server: kong/1.2.1
    < Content-Length: 434
    <
  • Connection #0 to host kong left intact
    {“next”:null,“data”:[{“id”:“e5e665cd-660d-4be7-88e4-db0dcaba8512”,“tags”:null,“paths”:["/new"],“destinations”:null,“protocols”:[“https”],“created_at”:1562196258,“snis”:null,“hosts”:[“frontkong-myhost-com-br”],“name”:“RT-NEW”,“preserve_host”:false,“regex_priority”:0,“strip_path”:true,“sources”:null,“updated_at”:1562196258,“https_redirect_status_code”:426,“service”:{“id”:“e247d460-0e17-456a-a23c-0f5d972b0700”},“methods”:[“GET”]}]}/ #

curl -vk …frontkong-myhost-com-br:8443/new

  • Trying 192.168.11.2:8443…
  • TCP_NODELAY set
  • Connected to frontkong-myhost-com-br (192.168.11.2) port 8443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: none
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: CN=*.myhost.com.br
  • start date: May 9 00:00:00 2018 GMT
  • expire date: Aug 11 00:00:00 2020 GMT
  • issuer: C=US; O=DigiCert Inc; OU=www-digicert-com; CN=RapidSSL RSA CA 2018
  • SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x564a8b073460)

GET /new HTTP/2
Host: frontkong-myhost-com-br:8443
User-Agent: curl/7.65.1
Accept: /

  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • old SSL session ID is stale, removing
  • Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
    < HTTP/2 404
    < date: Wed, 03 Jul 2019 20:28:27 GMT
    < content-type: application/json; charset=utf-8
    < content-length: 48
    < server: kong/1.2.1
    <
  • Connection #0 to host frontkong-myhost-com-br left intact

kong reload
Kong reloaded
/ #
/ #
/ #
/ # curl -vk …frontkong-myhost-com-br:8443/new

  • Trying 192.168.11.2:8443…
  • TCP_NODELAY set
  • Connected to frontkong-myhost-com-br (192.168.11.2) port 8443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: none
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: CN=*.myhost.com.br
  • start date: May 9 00:00:00 2018 GMT
  • expire date: Aug 11 00:00:00 2020 GMT
  • issuer: C=US; O=DigiCert Inc; OU=www-digicert-com; CN=RapidSSL RSA CA 2018
  • SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x5605d1e68460)

GET /new HTTP/2
Host: frontkong-myhost-com-br:8443
User-Agent: curl/7.65.1
Accept: /

  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • old SSL session ID is stale, removing
  • Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
    < HTTP/2 200
    < content-type: text/html;charset=utf-8
    < date: Wed, 03 Jul 2019 20:30:34 GMT
    < server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
    < x-frame-options: deny
    < x-xss-protection: 1
    < x-content-type-options: nosniff
    < content-security-policy: default-src ‘none’ ; script-src ‘self’ ‘unsafe-inline’ ‘unsafe-eval’ ; style-src ‘self’ ‘unsafe-inline’ ; img-src ‘self’ …media-rundeck-org ; font-src ‘self’ data: ; connect-src ‘self’ ; form-action ‘self’ ;
    < x-application-context: application:production:4440
    < expires: Thu, 01 Jan 1970 00:00:00 GMT
    < content-language: en-US
    < set-cookie: JSESSIONID=node0j2tzap8nyilhni8btuuosxme399.node0;Path=/;Secure;HttpOnly
    < x-kong-upstream-latency: 24
    < x-kong-proxy-latency: 2
    < via: kong/1.2.1
    <

##################################################################################################################

curl -v …8001/routes

  • Trying 192.168.12.2:8001…
  • TCP_NODELAY set
  • Connected to kong (192.168.12.2) port 8001 (#0)

GET /routes HTTP/1.1
Host: kong:8001
User-Agent: curl/7.65.1
Accept: /

  • Mark bundle as not supporting multiuse
    < HTTP/1.1 200 OK
    < Date: Wed, 03 Jul 2019 20:29:22 GMT
    < Content-Type: application/json; charset=utf-8
    < Connection: keep-alive
    < Access-Control-Allow-Origin: *
    < Server: kong/1.2.1
    < Content-Length: 434
    <
  • Connection #0 to host kong left intact
    {“next”:null,“data”:[{“id”:“e5e665cd-660d-4be7-88e4-db0dcaba8512”,“tags”:null,“paths”:["/new"],“destinations”:null,“protocols”:[“https”],“created_at”:1562196258,“snis”:null,“hosts”:[“frontkong-myhost-com-br”],“name”:“RT-NEW”,“preserve_host”:false,“regex_priority”:0,“strip_path”:true,“sources”:null,“updated_at”:1562196258,“https_redirect_status_code”:426,“service”:{“id”:“e247d460-0e17-456a-a23c-0f5d972b0700”},“methods”:[“GET”]}]}

curl -vk …frontkong-myhost-com-br:8443/new

  • Trying 192.168.12.2:8443…
  • TCP_NODELAY set
  • Connected to frontkong-myhost-com-br (192.168.12.2) port 8443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: none
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: CN=*.myhost.com.br
  • start date: May 9 00:00:00 2018 GMT
  • expire date: Aug 11 00:00:00 2020 GMT
  • issuer: C=US; O=DigiCert Inc; OU=www-digicert-com; CN=RapidSSL RSA CA 2018
  • SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x55caf3a74460)

GET /new HTTP/2
Host: frontkong-myhost-com-br:8443
User-Agent: curl/7.65.1
Accept: /

  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • old SSL session ID is stale, removing
  • Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
    < HTTP/2 404
    < date: Wed, 03 Jul 2019 20:29:44 GMT
    < content-type: application/json; charset=utf-8
    < content-length: 48
    < server: kong/1.2.1
    <
  • Connection #0 to host frontkong-myhost-com-br left intact
    {“message”:“no Route matched with those values”}

kong reload
Kong reloaded
/ #
/ #
/ # curl -vk …frontkong-myhost-com-br:8443/new

  • Trying 192.168.12.2:8443…
  • TCP_NODELAY set
  • Connected to frontkong-myhost-com-br (192.168.12.2) port 8443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: none
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: CN=*.myhost.com.br
  • start date: May 9 00:00:00 2018 GMT
  • expire date: Aug 11 00:00:00 2020 GMT
  • issuer: C=US; O=DigiCert Inc; OU=www-digicert-com; CN=RapidSSL RSA CA 2018
  • SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x557163d10460)

GET /new HTTP/2
Host: frontkong-myhost-com-br:8443
User-Agent: curl/7.65.1
Accept: /

  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • old SSL session ID is stale, removing
  • Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
    < HTTP/2 200
    < content-type: text/html;charset=utf-8
    < date: Wed, 03 Jul 2019 20:31:30 GMT
    < server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
    < x-frame-options: deny
    < x-xss-protection: 1
    < x-content-type-options: nosniff
    < content-security-policy: default-src ‘none’ ; script-src ‘self’ ‘unsafe-inline’ ‘unsafe-eval’ ; style-src ‘self’ ‘unsafe-inline’ ; img-src ‘self’ …media-rundeck-org ; font-src ‘self’ data: ; connect-src ‘self’ ; form-action ‘self’ ;
    < x-application-context: application:production:4440
    < expires: Thu, 01 Jan 1970 00:00:00 GMT
    < content-language: en-US
    < set-cookie: JSESSIONID=node01wxqs6n34ohgx3dc8cbteynoa400.node0;Path=/;Secure;HttpOnly
    < x-kong-upstream-latency: 21
    < x-kong-proxy-latency: 1
    < via: kong/1.2.1
    <

Regards,
Reginaldo

#5

Additionally, when I create route, the cluster_events table is not updated, in another environment that is working, it is updated:
kong=# select * from cluster_events ;
id | node_id | at | nbf | expire_at | channel | data
----±--------±—±----±----------±--------±-----
(0 rows)

#6

Here’s the solution: do not change the timezone of the containers.

I have identified error in the postgres container log:
docker logs compose_kong-database_1 -t
2019-07-04T21:45:43.449633091Z ERROR: relation “cluster_events” does not exist at character 116
2019-07-04T21:45:43.449682217Z STATEMENT: SELECT id, node_id, channel, data,
2019-07-04T21:45:43.449701337Z extract(epoch from at) as at,
2019-07-04T21:45:43.449728076Z extract(epoch from nbf) as nbf
2019-07-04T21:45:43.449755533Z FROM cluster_events
2019-07-04T21:45:43.449774333Z WHERE channel IN (‘invalidations’,‘balancer:targets’,‘balancer:post_health’,‘balancer:upstreams’,‘proxy-cache:purge’)
2019-07-04T21:45:43.449791153Z AND at > to_timestamp(1562276573.372000)
2019-07-04T21:45:43.449807591Z AND at <= to_timestamp(1562276743.434000)

Imagine what it might be, so I remembered that I had added timezone configuration in the containers to solve the wrong time problem in access.log.
After removing the lines below the docker-compose.yml and re-creating the containers the problem stopped:
volumes:

  • /etc/localtime:/etc/localtime:ro

You have now updated the cluster_events table:
kong=# select * from cluster_events ;
id | node_id | at | nbf | expire_at | channel
| data
--------------------------------------±-------------------------------------±---------------------------±----±---------------------------±-----------
—±------------------------------------------------
4d6a894e-66fc-4401-aedd-055076b51cff | b423c27e-d087-41ed-b903-7d1a64bdc287 | 2019-07-04 23:53:15.319+00 | | 2019-07-05 00:53:15.319+00 | invalidatio
ns | routes:db3a724d-6144-4b07-89b2-1d7869d8ec3f::::
68a1508e-60af-4919-a3f3-30a564b54919 | b423c27e-d087-41ed-b903-7d1a64bdc287 | 2019-07-04 23:53:15.323+00 | | 2019-07-05 00:53:15.323+00 | invalidatio
ns | router:version
(2 rows)

Now the routes are being invalidated / updated in the Kong B and C.

Thanks for the help dear.