Kong V14 - Kong error

Hi ,

We are running kong with cassandra (Networktopology,replicas:2) in single DC with K8.
when we start kong we are getting below error.

2018/07/30 11:31:43 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:206: could not load routes: [nil] could not execute page query: no host details for 10.244.5.154
stack traceback:
[C]: in function ‘assert’
/usr/local/share/lua/5.1/kong/init.lua:206: in function ‘init’
init_by_lua:3: in main chunk

@Niranjan_Kolly Hi,

To further help us debug this issue, could you share with us:

  1. The Kong configuration in use, ideally from the output of kong start -vv (which combines all of kong.conf, env variables, etc…)
  2. Make sure that there aren’t any other error or warning logs prior to this one

Thank you!

Hi,

we are not using kong.conf for the kong pods… All the env variable we are passing in the yaml which will overwrite the kong.conf .

Below are the parameters we are passing in yaml.

   env:
      - name: KONG_ADMIN_LISTEN
        value: "0.0.0.0:32001"
      - name: KONG_DATABASE
        value: cassandra
      - name: KONG_CASSANDRA_CONTACT_POINTS
        value: cassandra
      - name: KONG_CASSANDRA_KEYSPACE
        value: kong
      - name: KONG_CASSANDRA_USERNAME
        value: cassandra
      - name: KONG_CASSANDRA_PASSWORD
        value: cassandra
      - name: KONG_DB_UPDATE_PROPAGATION
        value: "5"
      - name: KONG_CASSANDRA_CONSISTENCY_LEVEL
        value: "ONE"
      - name: KONG_LOG_LEVEL
        value: debug
      - name: KONG_CASSANDRA_LB_POLICY
        value: RoundRobin
      - name: KONG_PROXY_ACCESS_LOG
        value: "/opt/log/kong/proxy_access"
      - name: KONG_ADMIN_ACCESS_LOG
        value: "/opt/log/kong/admin_access"
      - name: KONG_PROXY_ERROR_LOG
        value: "/opt/log/kong/proxy_error"
      - name: KONG_ADMIN_ERROR_LOG
        value: "/opt/log/kong/admin_error"

Below is the full error message from the kong start.

2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:431: init(): [dns-client] (re)configuring dns client
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:435: init(): [dns-client] staleTtl = 4
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:438: init(): [dns-client] noSynchronisation = false
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:457: init(): [dns-client] query order = LAST, SRV, A, CNAME
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:489: init(): [dns-client] adding A-record from ‘hosts’ file: kong-test-6dcbc5ff55-bt8m6 = 10.244.5.138
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:504: init(): [dns-client] adding AAAA-record from ‘hosts’ file: ip6-mcastprefix = [fe00::0]
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:504: init(): [dns-client] adding AAAA-record from ‘hosts’ file: ip6-localnet = [fe00::0]
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:489: init(): [dns-client] adding A-record from ‘hosts’ file: localhost = 127.0.0.1
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:504: init(): [dns-client] adding AAAA-record from ‘hosts’ file: localhost = [::1]
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:504: init(): [dns-client] adding AAAA-record from ‘hosts’ file: ip6-localhost = [::1]
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:504: init(): [dns-client] adding AAAA-record from ‘hosts’ file: ip6-loopback = [::1]
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:504: init(): [dns-client] adding AAAA-record from ‘hosts’ file: ip6-allnodes = [fe00::1]
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:504: init(): [dns-client] adding AAAA-record from ‘hosts’ file: ip6-allrouters = [fe00::2]
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:548: init(): [dns-client] nameserver 10.96.0.10
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:553: init(): [dns-client] attempts = 5
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:562: init(): [dns-client] timeout = 2000 ms
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:566: init(): [dns-client] ndots = 5
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:568: init(): [dns-client] search = default.svc.cluster.local, svc.cluster.local, cluster.local, vmware.com
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:574: init(): [dns-client] badTtl = 30 s
2018/08/02 06:49:45 [debug] 1#0: [lua] client.lua:576: init(): [dns-client] emptyTtl = 1 s
2018/08/02 06:49:45 [debug] 1#0: [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.244.6.131
2018/08/02 06:49:45 [debug] 1#0: [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.244.4.29
2018/08/02 06:49:45 [debug] 1#0: [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at cassandra
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:98: load_plugins(): Discovering used plugins
2018/08/02 06:49:45 [debug] 1#0: [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.244.6.131
2018/08/02 06:49:45 [debug] 1#0: [lua] cluster.lua:615: prepare(): [lua-cassandra] preparing SELECT * FROM plugins on host 10.244.6.131
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: response-transformer
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: oauth2
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: acl
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: ip-restriction
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: correlation-id
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: statsd
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: jwt
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: cors
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: request-transformer
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: basic-auth
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: key-auth
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: rate-limiting
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: file-log
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: http-log
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: tcp-log
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: hmac-auth
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: ldap-auth
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: datadog
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: mashery_oauth
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: zipkin
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: post-function
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: syslog
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: bot-detection
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: loggly
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: request-size-limiting
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: azure-functions
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: udp-log
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: response-ratelimiting
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: aws-lambda
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: prometheus
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: pre-function
2018/08/02 06:49:45 [debug] 1#0: [lua] init.lua:132: load_plugins(): Loading plugin: request-termination
2018/08/02 06:49:45 [debug] 1#0: [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.244.4.29
2018/08/02 06:49:45 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:206: could not load routes: [nil] could not execute page query: no host details for 10.244.5.127
stack traceback:
[C]: in function ‘assert’
/usr/local/share/lua/5.1/kong/init.lua:206: in function ‘init’
init_by_lua:3: in main chunk

Thanks,
Niranjan

Hi Guys

I hit the same issues with Kong 0.14 and Cassandra using values.yaml file from git Repo:

Original File: https://github.com/helm/charts/blob/master/stable/kong/values.yaml

Modifications:

Specify Kong configurations

Kong configurations guide https://getkong.org/docs/latest/configuration/

env:
database: cassandra
db_update_propagation: 1800

Cassandra chart configs

cassandra:
enabled: true
persistence:
enabled: false

PostgreSQL chart configs

postgresql:
enabled: false
postgresUser: kong
postgresDatabase: kong
persistence:
enabled: false

Errors:

[vagrant@localhost kong] kubectl get pods NAME READY STATUS RESTARTS AGE test-kingkong-cassandra-0 1/1 Running 0 18h test-kingkong-cassandra-1 1/1 Running 0 18h test-kingkong-cassandra-2 1/1 Running 0 17h test-kingkong-kong-5476d8b48f-7r496 0/1 CrashLoopBackOff 216 18h test-kingkong-kong-migrations-brnrj 0/1 Completed 5 18h tiller-deploy-f765fdbbf-s6hkt 1/1 Running 1 6d [vagrant@localhost kong] kubectl logs -f test-kingkong-kong-5476d8b48f-7r496
2018/08/30 21:03:46 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:206: could not load routes: no host details for 10.244.2.90
stack traceback:
[C]: in function ‘assert’
/usr/local/share/lua/5.1/kong/init.lua:206: in function ‘init’
init_by_lua:3: in main chunk
nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:206: could not load routes: no host details for 10.244.2.90
stack traceback:
[C]: in function ‘assert’
/usr/local/share/lua/5.1/kong/init.lua:206: in function ‘init’
init_by_lua:3: in main chunk
^C
[vagrant@localhost kong]$

Hi,

By 0.14, do you mean 0.14.0 or 0.14.1? Please be precise :slight_smile:

Sorry about that!! It was in values.yaml file…

image:
repository: kong
tag: 0.14.0
pullPolicy: IfNotPresent

Thanks

Chandra

Do you mind trying with 0.14.1? It should resolve this issue as noted in its release changelog.

Absolutely…will share the results shortly…

Appreciate quick reply.

Chandra

Yepp 0.14.1 does it!

[vagrant@localhost kong] kubectl get pods NAME READY STATUS RESTARTS AGE csr-kong-cassandra-0 1/1 Running 0 1h csr-kong-cassandra-1 1/1 Running 0 1h csr-kong-cassandra-2 1/1 Running 0 1h csr-kong-kong-85cf4859cb-lwflk 1/1 Running 5 1h csr-kong-kong-migrations-77nzc 0/1 Completed 5 1h [vagrant@localhost kong]

However, I see that kong had to restart 5 times waiting for cassandra to completely instantiate 3 instances…Is there a way to tell kong to wait until cassandra has done initializing.

Again appreciate the quick feedback on using 0.14.1…I can open a new ticket rather than piggie backing on same ticket…