Kong V14 - Kong error

Niranjan_Kolly · July 30, 2018, 11:51am

Hi ,

We are running kong with cassandra (Networktopology,replicas:2) in single DC with K8.
when we start kong we are getting below error.

2018/07/30 11:31:43 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:206: could not load routes: [nil] could not execute page query: no host details for 10.244.5.154
stack traceback:
[C]: in function ‘assert’
/usr/local/share/lua/5.1/kong/init.lua:206: in function ‘init’
init_by_lua:3: in main chunk

thibaultcha · August 1, 2018, 5:03pm

@Niranjan_Kolly Hi,

To further help us debug this issue, could you share with us:

The Kong configuration in use, ideally from the output of kong start -vv (which combines all of kong.conf, env variables, etc…)
Make sure that there aren’t any other error or warning logs prior to this one

Thank you!

Niranjan_Kolly · August 2, 2018, 7:54am

Hi,

we are not using kong.conf for the kong pods… All the env variable we are passing in the yaml which will overwrite the kong.conf .

Below are the parameters we are passing in yaml.

   env:
      - name: KONG_ADMIN_LISTEN
        value: "0.0.0.0:32001"
      - name: KONG_DATABASE
        value: cassandra
      - name: KONG_CASSANDRA_CONTACT_POINTS
        value: cassandra
      - name: KONG_CASSANDRA_KEYSPACE
        value: kong
      - name: KONG_CASSANDRA_USERNAME
        value: cassandra
      - name: KONG_CASSANDRA_PASSWORD
        value: cassandra
      - name: KONG_DB_UPDATE_PROPAGATION
        value: "5"
      - name: KONG_CASSANDRA_CONSISTENCY_LEVEL
        value: "ONE"
      - name: KONG_LOG_LEVEL
        value: debug
      - name: KONG_CASSANDRA_LB_POLICY
        value: RoundRobin
      - name: KONG_PROXY_ACCESS_LOG
        value: "/opt/log/kong/proxy_access"
      - name: KONG_ADMIN_ACCESS_LOG
        value: "/opt/log/kong/admin_access"
      - name: KONG_PROXY_ERROR_LOG
        value: "/opt/log/kong/proxy_error"
      - name: KONG_ADMIN_ERROR_LOG
        value: "/opt/log/kong/admin_error"

Below is the full error message from the kong start.

2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:431: init(): [dns-client] (re)configuring dns client
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:435: init(): [dns-client] staleTtl = 4
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:438: init(): [dns-client] noSynchronisation = false
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:457: init(): [dns-client] query order = LAST, SRV, A, CNAME
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:489: init(): [dns-client] adding A-record from ‘hosts’ file: kong-test-6dcbc5ff55-bt8m6 = 10.244.5.138
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:504: init(): [dns-client] adding AAAA-record from ‘hosts’ file: ip6-mcastprefix = [fe00::0]
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:504: init(): [dns-client] adding AAAA-record from ‘hosts’ file: ip6-localnet = [fe00::0]
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:489: init(): [dns-client] adding A-record from ‘hosts’ file: localhost = 127.0.0.1
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:504: init(): [dns-client] adding AAAA-record from ‘hosts’ file: localhost = [::1]
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:504: init(): [dns-client] adding AAAA-record from ‘hosts’ file: ip6-localhost = [::1]
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:504: init(): [dns-client] adding AAAA-record from ‘hosts’ file: ip6-loopback = [::1]
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:504: init(): [dns-client] adding AAAA-record from ‘hosts’ file: ip6-allnodes = [fe00::1]
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:504: init(): [dns-client] adding AAAA-record from ‘hosts’ file: ip6-allrouters = [fe00::2]
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:548: init(): [dns-client] nameserver 10.96.0.10
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:553: init(): [dns-client] attempts = 5
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:562: init(): [dns-client] timeout = 2000 ms
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:566: init(): [dns-client] ndots = 5
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:568: init(): [dns-client] search = default.svc.cluster.local, svc.cluster.local, cluster.local, vmware.com
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:574: init(): [dns-client] badTtl = 30 s
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:576: init(): [dns-client] emptyTtl = 1 s
2018/08/02 06:49:45 [debug] 1#0: [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.244.6.131
2018/08/02 06:49:45 [debug] 1#0: [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.244.4.29
2018/08/02 06:49:45 [debug] 1#0: [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at cassandra
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:98: load_plugins(): Discovering used plugins
2018/08/02 06:49:45 [debug] 1#0: [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.244.6.131
2018/08/02 06:49:45 [debug] 1#0: [lua] cluster.lua:615: prepare(): [lua-cassandra] preparing SELECT * FROM plugins on host 10.244.6.131
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: response-transformer
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: oauth2
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: acl
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: ip-restriction
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: correlation-id
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: statsd
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: jwt
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: cors
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: request-transformer
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: basic-auth
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: key-auth
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: rate-limiting
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: file-log
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: http-log
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: tcp-log
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: hmac-auth
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: ldap-auth
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: datadog
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: mashery_oauth
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: zipkin
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: post-function
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: syslog
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: bot-detection
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: loggly
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: request-size-limiting
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: azure-functions
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: udp-log
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: response-ratelimiting
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: aws-lambda
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: prometheus
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: pre-function
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: request-termination
2018/08/02 06:49:45 [debug] 1#0: [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.244.4.29
2018/08/02 06:49:45 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:206: could not load routes: [nil] could not execute page query: no host details for 10.244.5.127
stack traceback:
[C]: in function ‘assert’
/usr/local/share/lua/5.1/kong/init.lua:206: in function ‘init’
init_by_lua:3: in main chunk

Thanks,
Niranjan

chandrasrt · August 30, 2018, 9:14pm

Hi Guys

I hit the same issues with Kong 0.14 and Cassandra using values.yaml file from git Repo:

Original File: https://github.com/helm/charts/blob/master/stable/kong/values.yaml

Modifications:

Specify Kong configurations

Kong configurations guide https://getkong.org/docs/latest/configuration/

env:
database: cassandra
db_update_propagation: 1800

Cassandra chart configs

cassandra:
enabled: true
persistence:
enabled: false

PostgreSQL chart configs

postgresql:
enabled: false
postgresUser: kong
postgresDatabase: kong
persistence:
enabled: false

Errors:

[vagrant@localhost kong] kubectl get pods NAME READY STATUS RESTARTS AGE test-kingkong-cassandra-0 1/1 Running 0 18h test-kingkong-cassandra-1 1/1 Running 0 18h test-kingkong-cassandra-2 1/1 Running 0 17h test-kingkong-kong-5476d8b48f-7r496 0/1 CrashLoopBackOff 216 18h test-kingkong-kong-migrations-brnrj 0/1 Completed 5 18h tiller-deploy-f765fdbbf-s6hkt 1/1 Running 1 6d [vagrant@localhost kong] kubectl logs -f test-kingkong-kong-5476d8b48f-7r496
2018/08/30 21:03:46 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:206: could not load routes: no host details for 10.244.2.90
stack traceback:
[C]: in function ‘assert’
/usr/local/share/lua/5.1/kong/init.lua:206: in function ‘init’
init_by_lua:3: in main chunk
nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:206: could not load routes: no host details for 10.244.2.90
stack traceback:
[C]: in function ‘assert’
/usr/local/share/lua/5.1/kong/init.lua:206: in function ‘init’
init_by_lua:3: in main chunk
^C
[vagrant@localhost kong]$

thibaultcha · August 30, 2018, 9:18pm

Hi,

By 0.14, do you mean 0.14.0 or 0.14.1? Please be precise

chandrasrt · August 30, 2018, 9:20pm

Sorry about that!! It was in values.yaml file…

image:
repository: kong
tag: 0.14.0
pullPolicy: IfNotPresent

Thanks

Chandra

thibaultcha · August 30, 2018, 9:21pm

Do you mind trying with 0.14.1? It should resolve this issue as noted in its release changelog.

chandrasrt · August 30, 2018, 9:22pm

Absolutely…will share the results shortly…

Appreciate quick reply.

Chandra

chandrasrt · August 31, 2018, 4:18am

Yepp 0.14.1 does it!

[vagrant@localhost kong] kubectl get pods NAME READY STATUS RESTARTS AGE csr-kong-cassandra-0 1/1 Running 0 1h csr-kong-cassandra-1 1/1 Running 0 1h csr-kong-cassandra-2 1/1 Running 0 1h csr-kong-kong-85cf4859cb-lwflk 1/1 Running 5 1h csr-kong-kong-migrations-77nzc 0/1 Completed 5 1h [vagrant@localhost kong]

However, I see that kong had to restart 5 times waiting for cassandra to completely instantiate 3 instances…Is there a way to tell kong to wait until cassandra has done initializing.

Again appreciate the quick feedback on using 0.14.1…I can open a new ticket rather than piggie backing on same ticket…

Topic		Replies	Views
Kong 0.14.1 backed by Datastax Enterprise Cassandra 6.0.2 fails to start Questions	1	736	October 10, 2018
KONG cassandra error	1	547	February 8, 2019
Failed to parse host name "2a00:8a00:4000:20c::87:0fa6": invalid port) Questions	3	786	May 13, 2020
Could not start Kong, stopping services Questions	2	216	January 17, 2024
Plugin Cache invalidation problem Questions	6	901	January 14, 2019

Kong V14 - Kong error

Kong configurations guide https://getkong.org/docs/latest/configuration/

Cassandra chart configs

PostgreSQL chart configs

Related topics