Hello Dears,
Could anyone help me? There is no route invalidation occurring.
The DB_UPDATE_FREQUENCY is set to 5 seconds (default), I add a route on node A, the route works on node A, but on node B e C the route does not work.
I do GET curl http://kong:8001/routes on node B and the route added route_new appears, but when I do curl -k https://kong:8443/route_new the route does not work:
{“message”: “no route matched with those values”}.
I enabled the debug and the event below does not occur:
[lua] cluster_events.lua:248: [cluster_events] new event (channel: ‘invalidations’)
After manual kong reload on node B and C, the route works.
My scenery:
3 Kong CE v1.2.1 running on Docker connected to 1 Postgres 9.6
1 Konga v0.14.3 running on Docker.
Host 1: 1 Kong A, 1 Konga e 1 Postgres.
Host 2: 1 Kong B
Host 3: 1 Kong C
Communication between the Kongs and Postgres is working.
Can you show a response from curl http://kong:8001/routes
?
Might be better to keep your comments in the github issue: https://github.com/Kong/kong/issues/4764 , otherwise we are duplicating threads and eyes.
Hello Nareate,
I collected POST and GET in the 3 Kongs.
The “new” route POST was run on server 192.168.10.3, where it worked without kong reload the GET https://frontkong.myhost.com:8443/new.
On servers 192.168.11.2 and 192.168.12.2 GET https://frontkong.myhost.com:8443/new ran just after kong relaod.
Here are the details:
curl -v …8001/routes
- Trying 192.168.10.3:8001…
- TCP_NODELAY set
- Connected to kong (192.168.10.3) port 8001 (#0)
GET /routes HTTP/1.1
Host: kong:8001
User-Agent: curl/7.65.1
Accept: /
- Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 03 Jul 2019 20:31:27 GMT
< Content-Type: application/json; charset=utf-8
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Server: kong/1.2.1
< Content-Length: 434
< - Connection #0 to host kong left intact
{“next”:null,“data”:[{“id”:“e5e665cd-660d-4be7-88e4-db0dcaba8512”,“tags”:null,“paths”:["/new"],“destinations”:null,“protocols”:[“https”],“created_at”:1562196258,“snis”:null,“hosts”:[“frontkong-myhost-com-br”],“name”:“RT-NEW”,“preserve_host”:false,“regex_priority”:0,“strip_path”:true,“sources”:null,“updated_at”:1562196258,“https_redirect_status_code”:426,“service”:{“id”:“e247d460-0e17-456a-a23c-0f5d972b0700”},“methods”:[“GET”]}]}
curl -vk …8443/new
- Trying 192.168.10.3:8443…
- TCP_NODELAY set
- Connected to frontkong-myhost-com-br (192.168.10.3) port 8443 (#0)
- ALPN, offering h2
- ALPN, offering http/1.1
- successfully set certificate verify locations:
- CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none - TLSv1.3 (OUT), TLS handshake, Client hello (1):
- TLSv1.3 (IN), TLS handshake, Server hello (2):
- TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
- TLSv1.3 (IN), TLS handshake, Certificate (11):
- TLSv1.3 (IN), TLS handshake, CERT verify (15):
- TLSv1.3 (IN), TLS handshake, Finished (20):
- TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
- TLSv1.3 (OUT), TLS handshake, Finished (20):
- SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
- ALPN, server accepted to use h2
- Server certificate:
- subject: CN=*.myhost.com.br
- start date: May 9 00:00:00 2018 GMT
- expire date: Aug 11 00:00:00 2020 GMT
- issuer: C=US; O=DigiCert Inc; OU=www-digicert-com; CN=RapidSSL RSA CA 2018
- SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
- Using HTTP2, server supports multi-use
- Connection state changed (HTTP/2 confirmed)
- Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
- Using Stream ID: 1 (easy handle 0x55771bf91460)
GET /new HTTP/2
Host: frontkong-myhost-com-br:8443
User-Agent: curl/7.65.1
Accept: /
- TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
- TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
- old SSL session ID is stale, removing
- Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< content-type: text/html;charset=utf-8
< date: Wed, 03 Jul 2019 20:17:50 GMT
< server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
< x-frame-options: deny
< x-xss-protection: 1
< x-content-type-options: nosniff
< content-security-policy: default-src ‘none’ ; script-src ‘self’ ‘unsafe-inline’ ‘unsafe-eval’ ; style-src ‘self’ ‘unsafe-inline’ ; img-src ‘self’ …media-rundeck-org ; font-src ‘self’ data: ; connect-src ‘self’ ; form-action ‘self’ ;
< x-application-context: application:production:4440
< expires: Thu, 01 Jan 1970 00:00:00 GMT
< content-language: en-US
< set-cookie: JSESSIONID=node022e2njuu30lc1y1yxdmjsg1zt396.node0;Path=/;Secure;HttpOnly
< x-kong-upstream-latency: 22
< x-kong-proxy-latency: 5
< via: kong/1.2.1
<
##################################################################################################################
curl -v …8001/routes
- Trying 192.168.11.2:8001…
- TCP_NODELAY set
- Connected to kong (192.168.11.2) port 8001 (#0)
GET /routes HTTP/1.1
Host: kong:8001
User-Agent: curl/7.65.1
Accept: /
- Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 03 Jul 2019 20:28:10 GMT
< Content-Type: application/json; charset=utf-8
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Server: kong/1.2.1
< Content-Length: 434
< - Connection #0 to host kong left intact
{“next”:null,“data”:[{“id”:“e5e665cd-660d-4be7-88e4-db0dcaba8512”,“tags”:null,“paths”:["/new"],“destinations”:null,“protocols”:[“https”],“created_at”:1562196258,“snis”:null,“hosts”:[“frontkong-myhost-com-br”],“name”:“RT-NEW”,“preserve_host”:false,“regex_priority”:0,“strip_path”:true,“sources”:null,“updated_at”:1562196258,“https_redirect_status_code”:426,“service”:{“id”:“e247d460-0e17-456a-a23c-0f5d972b0700”},“methods”:[“GET”]}]}/ #
curl -vk …frontkong-myhost-com-br:8443/new
- Trying 192.168.11.2:8443…
- TCP_NODELAY set
- Connected to frontkong-myhost-com-br (192.168.11.2) port 8443 (#0)
- ALPN, offering h2
- ALPN, offering http/1.1
- successfully set certificate verify locations:
- CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none - TLSv1.3 (OUT), TLS handshake, Client hello (1):
- TLSv1.3 (IN), TLS handshake, Server hello (2):
- TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
- TLSv1.3 (IN), TLS handshake, Certificate (11):
- TLSv1.3 (IN), TLS handshake, CERT verify (15):
- TLSv1.3 (IN), TLS handshake, Finished (20):
- TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
- TLSv1.3 (OUT), TLS handshake, Finished (20):
- SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
- ALPN, server accepted to use h2
- Server certificate:
- subject: CN=*.myhost.com.br
- start date: May 9 00:00:00 2018 GMT
- expire date: Aug 11 00:00:00 2020 GMT
- issuer: C=US; O=DigiCert Inc; OU=www-digicert-com; CN=RapidSSL RSA CA 2018
- SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
- Using HTTP2, server supports multi-use
- Connection state changed (HTTP/2 confirmed)
- Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
- Using Stream ID: 1 (easy handle 0x564a8b073460)
GET /new HTTP/2
Host: frontkong-myhost-com-br:8443
User-Agent: curl/7.65.1
Accept: /
- TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
- TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
- old SSL session ID is stale, removing
- Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 404
< date: Wed, 03 Jul 2019 20:28:27 GMT
< content-type: application/json; charset=utf-8
< content-length: 48
< server: kong/1.2.1
< - Connection #0 to host frontkong-myhost-com-br left intact
kong reload
Kong reloaded
/ #
/ #
/ #
/ # curl -vk …frontkong-myhost-com-br:8443/new
- Trying 192.168.11.2:8443…
- TCP_NODELAY set
- Connected to frontkong-myhost-com-br (192.168.11.2) port 8443 (#0)
- ALPN, offering h2
- ALPN, offering http/1.1
- successfully set certificate verify locations:
- CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none - TLSv1.3 (OUT), TLS handshake, Client hello (1):
- TLSv1.3 (IN), TLS handshake, Server hello (2):
- TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
- TLSv1.3 (IN), TLS handshake, Certificate (11):
- TLSv1.3 (IN), TLS handshake, CERT verify (15):
- TLSv1.3 (IN), TLS handshake, Finished (20):
- TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
- TLSv1.3 (OUT), TLS handshake, Finished (20):
- SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
- ALPN, server accepted to use h2
- Server certificate:
- subject: CN=*.myhost.com.br
- start date: May 9 00:00:00 2018 GMT
- expire date: Aug 11 00:00:00 2020 GMT
- issuer: C=US; O=DigiCert Inc; OU=www-digicert-com; CN=RapidSSL RSA CA 2018
- SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
- Using HTTP2, server supports multi-use
- Connection state changed (HTTP/2 confirmed)
- Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
- Using Stream ID: 1 (easy handle 0x5605d1e68460)
GET /new HTTP/2
Host: frontkong-myhost-com-br:8443
User-Agent: curl/7.65.1
Accept: /
- TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
- TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
- old SSL session ID is stale, removing
- Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< content-type: text/html;charset=utf-8
< date: Wed, 03 Jul 2019 20:30:34 GMT
< server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
< x-frame-options: deny
< x-xss-protection: 1
< x-content-type-options: nosniff
< content-security-policy: default-src ‘none’ ; script-src ‘self’ ‘unsafe-inline’ ‘unsafe-eval’ ; style-src ‘self’ ‘unsafe-inline’ ; img-src ‘self’ …media-rundeck-org ; font-src ‘self’ data: ; connect-src ‘self’ ; form-action ‘self’ ;
< x-application-context: application:production:4440
< expires: Thu, 01 Jan 1970 00:00:00 GMT
< content-language: en-US
< set-cookie: JSESSIONID=node0j2tzap8nyilhni8btuuosxme399.node0;Path=/;Secure;HttpOnly
< x-kong-upstream-latency: 24
< x-kong-proxy-latency: 2
< via: kong/1.2.1
<
##################################################################################################################
curl -v …8001/routes
- Trying 192.168.12.2:8001…
- TCP_NODELAY set
- Connected to kong (192.168.12.2) port 8001 (#0)
GET /routes HTTP/1.1
Host: kong:8001
User-Agent: curl/7.65.1
Accept: /
- Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 03 Jul 2019 20:29:22 GMT
< Content-Type: application/json; charset=utf-8
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Server: kong/1.2.1
< Content-Length: 434
< - Connection #0 to host kong left intact
{“next”:null,“data”:[{“id”:“e5e665cd-660d-4be7-88e4-db0dcaba8512”,“tags”:null,“paths”:["/new"],“destinations”:null,“protocols”:[“https”],“created_at”:1562196258,“snis”:null,“hosts”:[“frontkong-myhost-com-br”],“name”:“RT-NEW”,“preserve_host”:false,“regex_priority”:0,“strip_path”:true,“sources”:null,“updated_at”:1562196258,“https_redirect_status_code”:426,“service”:{“id”:“e247d460-0e17-456a-a23c-0f5d972b0700”},“methods”:[“GET”]}]}
curl -vk …frontkong-myhost-com-br:8443/new
- Trying 192.168.12.2:8443…
- TCP_NODELAY set
- Connected to frontkong-myhost-com-br (192.168.12.2) port 8443 (#0)
- ALPN, offering h2
- ALPN, offering http/1.1
- successfully set certificate verify locations:
- CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none - TLSv1.3 (OUT), TLS handshake, Client hello (1):
- TLSv1.3 (IN), TLS handshake, Server hello (2):
- TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
- TLSv1.3 (IN), TLS handshake, Certificate (11):
- TLSv1.3 (IN), TLS handshake, CERT verify (15):
- TLSv1.3 (IN), TLS handshake, Finished (20):
- TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
- TLSv1.3 (OUT), TLS handshake, Finished (20):
- SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
- ALPN, server accepted to use h2
- Server certificate:
- subject: CN=*.myhost.com.br
- start date: May 9 00:00:00 2018 GMT
- expire date: Aug 11 00:00:00 2020 GMT
- issuer: C=US; O=DigiCert Inc; OU=www-digicert-com; CN=RapidSSL RSA CA 2018
- SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
- Using HTTP2, server supports multi-use
- Connection state changed (HTTP/2 confirmed)
- Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
- Using Stream ID: 1 (easy handle 0x55caf3a74460)
GET /new HTTP/2
Host: frontkong-myhost-com-br:8443
User-Agent: curl/7.65.1
Accept: /
- TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
- TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
- old SSL session ID is stale, removing
- Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 404
< date: Wed, 03 Jul 2019 20:29:44 GMT
< content-type: application/json; charset=utf-8
< content-length: 48
< server: kong/1.2.1
< - Connection #0 to host frontkong-myhost-com-br left intact
{“message”:“no Route matched with those values”}
kong reload
Kong reloaded
/ #
/ #
/ # curl -vk …frontkong-myhost-com-br:8443/new
- Trying 192.168.12.2:8443…
- TCP_NODELAY set
- Connected to frontkong-myhost-com-br (192.168.12.2) port 8443 (#0)
- ALPN, offering h2
- ALPN, offering http/1.1
- successfully set certificate verify locations:
- CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none - TLSv1.3 (OUT), TLS handshake, Client hello (1):
- TLSv1.3 (IN), TLS handshake, Server hello (2):
- TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
- TLSv1.3 (IN), TLS handshake, Certificate (11):
- TLSv1.3 (IN), TLS handshake, CERT verify (15):
- TLSv1.3 (IN), TLS handshake, Finished (20):
- TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
- TLSv1.3 (OUT), TLS handshake, Finished (20):
- SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
- ALPN, server accepted to use h2
- Server certificate:
- subject: CN=*.myhost.com.br
- start date: May 9 00:00:00 2018 GMT
- expire date: Aug 11 00:00:00 2020 GMT
- issuer: C=US; O=DigiCert Inc; OU=www-digicert-com; CN=RapidSSL RSA CA 2018
- SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
- Using HTTP2, server supports multi-use
- Connection state changed (HTTP/2 confirmed)
- Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
- Using Stream ID: 1 (easy handle 0x557163d10460)
GET /new HTTP/2
Host: frontkong-myhost-com-br:8443
User-Agent: curl/7.65.1
Accept: /
- TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
- TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
- old SSL session ID is stale, removing
- Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< content-type: text/html;charset=utf-8
< date: Wed, 03 Jul 2019 20:31:30 GMT
< server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
< x-frame-options: deny
< x-xss-protection: 1
< x-content-type-options: nosniff
< content-security-policy: default-src ‘none’ ; script-src ‘self’ ‘unsafe-inline’ ‘unsafe-eval’ ; style-src ‘self’ ‘unsafe-inline’ ; img-src ‘self’ …media-rundeck-org ; font-src ‘self’ data: ; connect-src ‘self’ ; form-action ‘self’ ;
< x-application-context: application:production:4440
< expires: Thu, 01 Jan 1970 00:00:00 GMT
< content-language: en-US
< set-cookie: JSESSIONID=node01wxqs6n34ohgx3dc8cbteynoa400.node0;Path=/;Secure;HttpOnly
< x-kong-upstream-latency: 21
< x-kong-proxy-latency: 1
< via: kong/1.2.1
<
Regards,
Reginaldo
Additionally, when I create route, the cluster_events table is not updated, in another environment that is working, it is updated:
kong=# select * from cluster_events ;
id | node_id | at | nbf | expire_at | channel | data
----±--------±—±----±----------±--------±-----
(0 rows)
Here’s the solution: do not change the timezone of the containers.
I have identified error in the postgres container log:
docker logs compose_kong-database_1 -t
2019-07-04T21:45:43.449633091Z ERROR: relation “cluster_events” does not exist at character 116
2019-07-04T21:45:43.449682217Z STATEMENT: SELECT id, node_id, channel, data,
2019-07-04T21:45:43.449701337Z extract(epoch from at) as at,
2019-07-04T21:45:43.449728076Z extract(epoch from nbf) as nbf
2019-07-04T21:45:43.449755533Z FROM cluster_events
2019-07-04T21:45:43.449774333Z WHERE channel IN (‘invalidations’,‘balancer:targets’,‘balancer:post_health’,‘balancer:upstreams’,‘proxy-cache:purge’)
2019-07-04T21:45:43.449791153Z AND at > to_timestamp(1562276573.372000)
2019-07-04T21:45:43.449807591Z AND at <= to_timestamp(1562276743.434000)
Imagine what it might be, so I remembered that I had added timezone configuration in the containers to solve the wrong time problem in access.log.
After removing the lines below the docker-compose.yml and re-creating the containers the problem stopped:
volumes:
- /etc/localtime:/etc/localtime:ro
You have now updated the cluster_events table:
kong=# select * from cluster_events ;
id | node_id | at | nbf | expire_at | channel
| data
--------------------------------------±-------------------------------------±---------------------------±----±---------------------------±-----------
—±------------------------------------------------
4d6a894e-66fc-4401-aedd-055076b51cff | b423c27e-d087-41ed-b903-7d1a64bdc287 | 2019-07-04 23:53:15.319+00 | | 2019-07-05 00:53:15.319+00 | invalidatio
ns | routes:db3a724d-6144-4b07-89b2-1d7869d8ec3f::::
68a1508e-60af-4919-a3f3-30a564b54919 | b423c27e-d087-41ed-b903-7d1a64bdc287 | 2019-07-04 23:53:15.323+00 | | 2019-07-05 00:53:15.323+00 | invalidatio
ns | router:version
(2 rows)
Now the routes are being invalidated / updated in the Kong B and C.
Thanks for the help dear.