Run Kong as Kubernetes Service with IPv6 Support

smuthunoori · March 10, 2020, 10:41pm

Hi,
I am trying to run Kong 0.14.1 as a Kubernetes Deployment and access it through the Kubernetes Service using DNS. My cluster uses IPv6 addresses. However, I am having difficulty in running commands like “Kong migrations list”. I am using Cassandra DB and is running as a K8 Service as well and resolves through DNS name.

I see the following error:(when I use IPv6 address instead of DNS name as cassandra_contact_points)

Error:
/usr/local/share/lua/5.1/kong/cmd/migrations.lua:37: [cassandra error] all hosts tried for query failed. 3ffd::9755: host seems unhealthy, considering it down (Address family for hostname not supported)
stack traceback:
[C]: in function ‘assert’
/usr/local/share/lua/5.1/kong/cmd/migrations.lua:37: in function ‘cmd_exec’
/usr/local/share/lua/5.1/kong/cmd/init.lua:87: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:87>
[C]: in function ‘xpcall’
/usr/local/share/lua/5.1/kong/cmd/init.lua:87: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:44>
/usr/local/bin/kong:7: in function ‘file_gen’
init_worker_by_lua:48: in function <init_worker_by_lua:46>
[C]: in function ‘xpcall’
init_worker_by_lua:55: in function <init_worker_by_lua:53>

I see the following error: (when I use DNS name as cassandra_contact_points)

Error:
/usr/local/share/lua/5.1/kong/cmd/migrations.lua:37: [cassandra error] all hosts tried for query failed. cassandra.maglev-system.svc.cluster.local: host seems unhealthy, considering it down (host or service not provided, or not known)
stack traceback:
[C]: in function ‘assert’
/usr/local/share/lua/5.1/kong/cmd/migrations.lua:37: in function ‘cmd_exec’
/usr/local/share/lua/5.1/kong/cmd/init.lua:87: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:87>
[C]: in function ‘xpcall’
/usr/local/share/lua/5.1/kong/cmd/init.lua:87: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:44>
/usr/local/bin/kong:7: in function ‘file_gen’
init_worker_by_lua:48: in function <init_worker_by_lua:46>
[C]: in function ‘xpcall’
init_worker_by_lua:55: in function <init_worker_by_lua:53>

Here is the output from my “netcat” command which resolves the DNS name to the right IPv6 address:

root@maglev-master-20010420:/# nslookup -query=AAAA cassandra.maglev-system.svc.cluster.local
Server: 2001:10::1
Address: 2001:10::1#53

cassandra.maglev-system.svc.cluster.local has AAAA address 3ffd::9755

Am I missing something? Please help!

smuthunoori · April 4, 2020, 4:39pm

For benefit of others who might be running into similar problem, this is how I was able to resolve:

Modify the cassandra/socket.lua to use tcp6() socket. In my environment, I have an env variable that would help me to identify if the addressing is IPv6 or IPv4 enabled. Hence used the same as shown below (marked with ** below):

do
  local setmetatable = setmetatable

  if ngx then
    local log, WARN, INFO = ngx.log, ngx.WARN, ngx.INFO
    local get_phase = ngx.get_phase

    function _M.tcp(...)
      local phase = get_phase()
      if not forced_luasocket_phases[phase]
         and COSOCKET_PHASES[phase]
         or forbidden_luasocket_phases[phase] then
        return ngx.socket.tcp(...)
      end

      -- LuaSocket
      if phase ~= 'init' then
        if forced_luasocket_phases[phase] then
          log(INFO, 'support for cosocket in this context, but LuaSocket forced')
        else
          log(WARN, 'no support for cosockets in this context, falling back to LuaSocket')
        end
      end

      local socket = require 'socket'
      **if os.getenv("ADDRESS_FAMILY") == "ipv6" then**
**        local sock = socket.tcp6(...)**
        return setmetatable({
          sock = sock
        }, proxy_mt)
      else
        local sock = socket.tcp(...)
        return setmetatable({
          sock = sock
        }, proxy_mt)
      end
    end
  else
    function _M.tcp(...)
      local socket = require 'socket'
      **if os.getenv("ADDRESS_FAMILY") == "ipv6" then**
**        local sock = socket.tcp6(...)**
        return setmetatable({
          sock = sock
        }, proxy_mt)
      else
        local sock = socket.tcp(...)
        return setmetatable({
          sock = sock
        }, proxy_mt)
      end
    end
  end
end

angrosz · April 8, 2020, 8:51pm

I have exactly the same problem as you described but in Kong 1.1.1. I modified the cassandra/socket.lua to use tcp6() socket as you suggested but it didn’t help. Did you do something additional?

bash-4.2$ getent hosts ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local
2001:283:4000:2001::5167 ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local

I’d appreciate your help!

smuthunoori · April 10, 2020, 3:24am

Can you please share me the output of “kong migrations list” command? I am assuming that you have the following:

DNS resolver configured both in /etc/resolv.conf and kong.conf.
cassandra_contact_points is using the DNS name of your cassandra svc
nslookup command from your Kong container is able to resolve the cassandra svc DNS name.

Please confirm.

angrosz · April 10, 2020, 7:15am

@smuthunoori
I’ve opened an issue https://github.com/thibaultcha/lua-cassandra/issues/139#issue-596498141
I described in details my problem.
Regarding your 3 assumptions I’ll check the configuration and let you know.

Best Regards,
Ania

angrosz · April 10, 2020, 5:36pm

I configured Kong via environment variables. Here are KONG_* variables:

KONG_CASSANDRA_CONSISTENCY=QUORUM
KONG_CASSANDRA_STATEFUL_SET=-ccas-apache
KONG_CASSANDRA_REPL_FACTOR=1
KONG_CASSANDRA_USERNAME=suadmincass
KONG_LARGE_CLIENT_HEADER_BUFFERS_SIZE=8k
KONG_DATABASE=cassandra
KONG_DB_UPDATE_PROPAGATION=1
KONG_CASSANDRA_PASSWORD=z6l0ouLVmgonkfua
KONG_NGINX_DAEMON=off
KONG_CLIENT_HEADER_BUFFER_SIZE=1k
KONG_ANONYMOUS_REPORTS=off
KONG_LARGE_CLIENT_HEADER_BUFFERS_NUMBER=4
KONG_CASSANDRA_SCHEMA_CONSENSUS_TIMEOUT=180000
KONG_CASSANDRA_CONTACT_POINTS=ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local

Ad.1
I found a ticket

I modified /usr/local/share/lua/5.1/kong/conf_loader.lua file (according to this issue) and I exported additional variable:
KONG_DNS_ORDER=LAST,SRV,A,CNAME,AAAA

my /etc/resolv.conf file is following:

bash-4.2$ cat /etc/resolv.conf
nameserver fc00::13
search csfapiexp-ccas.svc.cluster.local csfapiexp-ckng.svc.cluster.local svc.cluster.local cluster.local bcmt.cluster.local
options ndots:5

Ad.2

bash-4.2$ getent hosts ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local
2001:283:4000:2001::5167 ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local

Ad.3

bash-4.2$ nslookup ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local
Server:         fc00::13
Address:        fc00::13#53

Name:   ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local
Address: 2001:283:4000:2001::5167

bash-4.2$ dig AAAA ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> AAAA ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44480
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local. IN AAAA

;; ANSWER SECTION:
ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local. 5 IN AAAA 2001:283:4000:2001::5167

;; Query time: 11 msec
;; SERVER: fc00::13#53(fc00::13)
;; WHEN: Fri Apr 10 17:35:49 UTC 2020
;; MSG SIZE  rcvd: 155

smuthunoori · April 10, 2020, 6:16pm

Ania,
Yes, the conf_loader.lua was another change I had to make. Besides this I also had to make change similar to cassandra/socket.lua in socket/http.lua to use tcp6(). These four changes helped me to resolve the issue. Hope it helps!

File paths:
/usr/local/share/lua/5.1/kong/conf_loader.lua
/usr/local/share/lua/5.1/cassandra/socket.lua
/usr/local/share/lua/5.1/socket/http.lua

My first step was to get command “kong migrations list” working. This did not require the conf_loader and socket/http.lua changes. However, when I started kong, then I had to make these two changes as well.

Regards

angrosz · April 10, 2020, 11:49pm

I’m using Kong 1.1.1

I did the following modifications:

export KONG_DNS_ORDER=LAST,SRV,A,CNAME,AAAA
1.1. changes in /usr/local/share/lua/5.1/kong/conf_loader.lua
export ADDRESS_FAMILY=ipv6
2.1. changes in /usr/local/share/lua/5.1/cassandra/socket.lua

FIRST SCENARIO
KONG_CASSANDRA_CONTACT_POINTS=ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local

bash-4.2$ nslookup ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local
Server:         fc00::13
Address:        fc00::13#53

Name:   ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local
Address: 2001:283:4000:2001::5167

bash-4.2$ dig AAAA ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> AAAA ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44480
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local. IN AAAA

;; ANSWER SECTION:
ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local. 5 IN AAAA 2001:283:4000:2001::5167

;; Query time: 11 msec
;; SERVER: fc00::13#53(fc00::13)
;; WHEN: Fri Apr 10 17:35:49 UTC 2020
;; MSG SIZE  rcvd: 155

bash-4.2$ kong migrations list --vv
2020/04/11 00:01:49 [verbose] Kong: 1.1.1
2020/04/11 00:01:49 [debug] ngx_lua: 10013
2020/04/11 00:01:49 [debug] nginx: 1013006
2020/04/11 00:01:49 [debug] Lua: LuaJIT 2.1.0-beta3
2020/04/11 00:01:49 [verbose] no config file found at /etc/kong/kong.conf
2020/04/11 00:01:49 [verbose] no config file, skip loading
2020/04/11 00:01:49 [debug] reading environment variables
2020/04/11 00:01:49 [debug] KONG_CASSANDRA_USERNAME ENV found with "suadmincass"
2020/04/11 00:01:49 [debug] KONG_CASSANDRA_PASSWORD ENV found with "******"
2020/04/11 00:01:49 [debug] KONG_DB_UPDATE_PROPAGATION ENV found with "1"
2020/04/11 00:01:49 [debug] KONG_DNS_ORDER ENV found with "LAST,SRV,A,CNAME,AAAA"
2020/04/11 00:01:49 [debug] KONG_DATABASE ENV found with "cassandra"
2020/04/11 00:01:49 [debug] KONG_LARGE_CLIENT_HEADER_BUFFERS_NUMBER ENV found with "4"
2020/04/11 00:01:49 [debug] KONG_CLIENT_HEADER_BUFFER_SIZE ENV found with "1k"
2020/04/11 00:01:49 [debug] KONG_CASSANDRA_SCHEMA_CONSENSUS_TIMEOUT ENV found with "180000"
2020/04/11 00:01:49 [debug] KONG_CASSANDRA_CONTACT_POINTS ENV found with "ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local"
2020/04/11 00:01:49 [debug] KONG_LARGE_CLIENT_HEADER_BUFFERS_SIZE ENV found with "8k"
2020/04/11 00:01:49 [debug] KONG_CASSANDRA_REPL_FACTOR ENV found with "1"
2020/04/11 00:01:49 [debug] KONG_NGINX_DAEMON ENV found with "off"
2020/04/11 00:01:49 [debug] KONG_ANONYMOUS_REPORTS ENV found with "off"
2020/04/11 00:01:49 [debug] KONG_CASSANDRA_CONSISTENCY ENV found with "QUORUM"
2020/04/11 00:01:49 [debug] admin_access_log = "logs/admin_access.log"
2020/04/11 00:01:49 [debug] admin_api_spec_listen = {"0.0.0.0:8080","0.0.0.0:9443 ssl"}
2020/04/11 00:01:49 [debug] admin_error_log = "logs/error.log"
2020/04/11 00:01:49 [debug] admin_listen = {"0.0.0.0:8001","0.0.0.0:8444 ssl"}
2020/04/11 00:01:49 [debug] anonymous_reports = false
2020/04/11 00:01:49 [debug] cache_negative_ttl_default = 60
2020/04/11 00:01:49 [debug] cache_negative_ttl_max = 300
2020/04/11 00:01:49 [debug] cache_negative_ttl_min = 1
2020/04/11 00:01:49 [debug] cache_ttl_default = 3600
2020/04/11 00:01:49 [debug] cache_ttl_max = 3600
2020/04/11 00:01:49 [debug] cache_ttl_min = 60
2020/04/11 00:01:49 [debug] cassandra_cluster_refresh_interval = 15
2020/04/11 00:01:49 [debug] cassandra_consistency = "QUORUM"
2020/04/11 00:01:49 [debug] cassandra_contact_points = {"ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local"}
2020/04/11 00:01:49 [debug] cassandra_data_centers = {"dc1:2","dc2:3"}
2020/04/11 00:01:49 [debug] cassandra_keyspace = "kong"
2020/04/11 00:01:49 [debug] cassandra_lb_policy = "RequestRoundRobin"
2020/04/11 00:01:49 [debug] cassandra_password = "******"
2020/04/11 00:01:49 [debug] cassandra_port = 9042
2020/04/11 00:01:49 [debug] cassandra_repl_factor = 1
2020/04/11 00:01:49 [debug] cassandra_repl_strategy = "SimpleStrategy"
2020/04/11 00:01:49 [debug] cassandra_schema_consensus_timeout = 180000
2020/04/11 00:01:49 [debug] cassandra_ssl = false
2020/04/11 00:01:49 [debug] cassandra_ssl_verify = false
2020/04/11 00:01:49 [debug] cassandra_timeout = 5000
2020/04/11 00:01:49 [debug] cassandra_username = "suadmincass"
2020/04/11 00:01:49 [debug] client_body_buffer_size = "8k"
2020/04/11 00:01:49 [debug] client_header_buffer_size = "1k"
2020/04/11 00:01:49 [debug] client_max_body_size = "0"
2020/04/11 00:01:49 [debug] client_ssl = false
2020/04/11 00:01:49 [debug] database = "cassandra"
2020/04/11 00:01:49 [debug] db_cache_ttl = 0
2020/04/11 00:01:49 [debug] db_resurrect_ttl = 30
2020/04/11 00:01:49 [debug] db_update_frequency = 5
2020/04/11 00:01:49 [debug] db_update_propagation = 1
2020/04/11 00:01:49 [debug] dns_error_ttl = 1
2020/04/11 00:01:49 [debug] dns_hostsfile = "/etc/hosts"
2020/04/11 00:01:49 [debug] dns_no_sync = false
2020/04/11 00:01:49 [debug] dns_not_found_ttl = 30
2020/04/11 00:01:49 [debug] dns_order = {"LAST","SRV","A","CNAME","AAAA"}
2020/04/11 00:01:49 [debug] dns_resolver = {}
2020/04/11 00:01:49 [debug] dns_stale_ttl = 4
2020/04/11 00:01:49 [debug] error_default_type = "text/plain"
2020/04/11 00:01:49 [debug] headers = {"server_tokens","latency_tokens"}
2020/04/11 00:01:49 [debug] large_client_header_buffers_number = 4
2020/04/11 00:01:49 [debug] large_client_header_buffers_size = "8k"
2020/04/11 00:01:49 [debug] log_anonymize = "OFF"
2020/04/11 00:01:49 [debug] log_level = "notice"
2020/04/11 00:01:49 [debug] lru_cache_size = 500000
2020/04/11 00:01:49 [debug] lua_package_cpath = ""
2020/04/11 00:01:49 [debug] lua_package_path = "./?.lua;./?/init.lua;"
2020/04/11 00:01:49 [debug] lua_socket_pool_size = 30
2020/04/11 00:01:49 [debug] lua_ssl_verify_depth = 1
2020/04/11 00:01:49 [debug] maria_database = "kong"
2020/04/11 00:01:49 [debug] maria_host = "127.0.0.1"
2020/04/11 00:01:49 [debug] maria_password = "******"
2020/04/11 00:01:49 [debug] maria_port = 3306
2020/04/11 00:01:49 [debug] maria_socket_keepalive_timeout = 60000
2020/04/11 00:01:49 [debug] maria_socket_pool_size = 20
2020/04/11 00:01:49 [debug] maria_ssl = false
2020/04/11 00:01:49 [debug] maria_ssl_verify = false
2020/04/11 00:01:49 [debug] maria_timeout = 10000
2020/04/11 00:01:49 [debug] maria_user = "kong"
2020/04/11 00:01:49 [debug] mem_cache_size = "128m"
2020/04/11 00:01:49 [debug] nginx_admin_directives = {}
2020/04/11 00:01:49 [debug] nginx_daemon = "off"
2020/04/11 00:01:49 [debug] nginx_http_directives = {}
2020/04/11 00:01:49 [debug] nginx_optimizations = true
2020/04/11 00:01:49 [debug] nginx_proxy_directives = {}
2020/04/11 00:01:49 [debug] nginx_sproxy_directives = {}
2020/04/11 00:01:49 [debug] nginx_stream_directives = {}
2020/04/11 00:01:49 [debug] nginx_user = "nobody nobody"
2020/04/11 00:01:49 [debug] nginx_worker_processes = "auto"
2020/04/11 00:01:49 [debug] origins = {}
2020/04/11 00:01:49 [debug] pg_database = "kong"
2020/04/11 00:01:49 [debug] pg_host = "127.0.0.1"
2020/04/11 00:01:49 [debug] pg_port = 5432
2020/04/11 00:01:49 [debug] pg_ssl = false
2020/04/11 00:01:49 [debug] pg_ssl_verify = false
2020/04/11 00:01:49 [debug] pg_timeout = 5000
2020/04/11 00:01:49 [debug] pg_user = "kong"
2020/04/11 00:01:49 [debug] plugins = {"bundled"}
2020/04/11 00:01:49 [debug] prefix = "/usr/local/kong/"
2020/04/11 00:01:49 [debug] proxy_access_log = "logs/access.log"
2020/04/11 00:01:49 [debug] proxy_error_log = "logs/error.log"
2020/04/11 00:01:49 [debug] proxy_listen = {"0.0.0.0:8000","0.0.0.0:8443 ssl"}
2020/04/11 00:01:49 [debug] real_ip_header = "X-Real-IP"
2020/04/11 00:01:49 [debug] real_ip_recursive = "off"
2020/04/11 00:01:49 [debug] ssl_cipher_suite = "modern"
2020/04/11 00:01:49 [debug] ssl_ciphers = "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256"
2020/04/11 00:01:49 [debug] stream_listen = {"off"}
2020/04/11 00:01:49 [debug] trusted_ips = {}
2020/04/11 00:01:49 [debug] upstream_keepalive = 60
2020/04/11 00:01:49 [debug] validate_certs_refresh_interval = 300
2020/04/11 00:01:49 [debug] validate_certs_warn_before = 7
2020/04/11 00:01:49 [verbose] prefix in use: /usr/local/kong
2020/04/11 00:01:49 [error] could not resolve Cassandra contact point 'ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local': failed to create a resolver: failed to set peer name: host or service not provided, or not known
Error:
./kong/cmd/migrations.lua:106: could not resolve any of the provided Cassandra contact points (cassandra_contact_points = 'ccas-ccas-apache.csfapiexp-ccas.svc.cluster.local')
stack traceback:
        [C]: in function 'assert'
        ./kong/cmd/migrations.lua:106: in function 'cmd_exec'
        ./kong/cmd/init.lua:90: in function <./kong/cmd/init.lua:90>
        [C]: in function 'xpcall'
        ./kong/cmd/init.lua:90: in function <./kong/cmd/init.lua:47>
        /etc/kong/openresty_temp_script.lua:6: in function 'file_gen'
        init_worker_by_lua:49: in function <init_worker_by_lua:47>
        [C]: in function 'xpcall'
        init_worker_by_lua:56: in function <init_worker_by_lua:54>
bash-4.2$

SECOND SCENARIO
KONG_CASSANDRA_CONTACT_POINTS=2001:283:4000:2001::5167

bash-4.2$ kong migrations list --vv
2020/04/11 00:05:32 [verbose] Kong: 1.1.1
2020/04/11 00:05:32 [debug] ngx_lua: 10013
2020/04/11 00:05:32 [debug] nginx: 1013006
2020/04/11 00:05:32 [debug] Lua: LuaJIT 2.1.0-beta3
2020/04/11 00:05:32 [verbose] no config file found at /etc/kong/kong.conf
2020/04/11 00:05:32 [verbose] no config file, skip loading
2020/04/11 00:05:32 [debug] reading environment variables
2020/04/11 00:05:32 [debug] KONG_CASSANDRA_USERNAME ENV found with "suadmincass"
2020/04/11 00:05:32 [debug] KONG_CASSANDRA_PASSWORD ENV found with "******"
2020/04/11 00:05:32 [debug] KONG_DB_UPDATE_PROPAGATION ENV found with "1"
2020/04/11 00:05:32 [debug] KONG_DNS_ORDER ENV found with "LAST,SRV,A,CNAME,AAAA"
2020/04/11 00:05:32 [debug] KONG_DATABASE ENV found with "cassandra"
2020/04/11 00:05:32 [debug] KONG_LARGE_CLIENT_HEADER_BUFFERS_NUMBER ENV found with "4"
2020/04/11 00:05:32 [debug] KONG_CLIENT_HEADER_BUFFER_SIZE ENV found with "1k"
2020/04/11 00:05:32 [debug] KONG_CASSANDRA_SCHEMA_CONSENSUS_TIMEOUT ENV found with "180000"
2020/04/11 00:05:32 [debug] KONG_CASSANDRA_CONTACT_POINTS ENV found with "2001:283:4000:2001::5167"
2020/04/11 00:05:32 [debug] KONG_LARGE_CLIENT_HEADER_BUFFERS_SIZE ENV found with "8k"
2020/04/11 00:05:32 [debug] KONG_CASSANDRA_REPL_FACTOR ENV found with "1"
2020/04/11 00:05:32 [debug] KONG_NGINX_DAEMON ENV found with "off"
2020/04/11 00:05:32 [debug] KONG_ANONYMOUS_REPORTS ENV found with "off"
2020/04/11 00:05:32 [debug] KONG_CASSANDRA_CONSISTENCY ENV found with "QUORUM"
2020/04/11 00:05:32 [debug] admin_access_log = "logs/admin_access.log"
2020/04/11 00:05:32 [debug] admin_api_spec_listen = {"0.0.0.0:8080","0.0.0.0:9443 ssl"}
2020/04/11 00:05:32 [debug] admin_error_log = "logs/error.log"
2020/04/11 00:05:32 [debug] admin_listen = {"0.0.0.0:8001","0.0.0.0:8444 ssl"}
2020/04/11 00:05:32 [debug] anonymous_reports = false
2020/04/11 00:05:32 [debug] cache_negative_ttl_default = 60
2020/04/11 00:05:32 [debug] cache_negative_ttl_max = 300
2020/04/11 00:05:32 [debug] cache_negative_ttl_min = 1
2020/04/11 00:05:32 [debug] cache_ttl_default = 3600
2020/04/11 00:05:32 [debug] cache_ttl_max = 3600
2020/04/11 00:05:32 [debug] cache_ttl_min = 60
2020/04/11 00:05:32 [debug] cassandra_cluster_refresh_interval = 15
2020/04/11 00:05:32 [debug] cassandra_consistency = "QUORUM"
2020/04/11 00:05:32 [debug] cassandra_contact_points = {"2001:283:4000:2001::5167"}
2020/04/11 00:05:32 [debug] cassandra_data_centers = {"dc1:2","dc2:3"}
2020/04/11 00:05:32 [debug] cassandra_keyspace = "kong"
2020/04/11 00:05:32 [debug] cassandra_lb_policy = "RequestRoundRobin"
2020/04/11 00:05:32 [debug] cassandra_password = "******"
2020/04/11 00:05:32 [debug] cassandra_port = 9042
2020/04/11 00:05:32 [debug] cassandra_repl_factor = 1
2020/04/11 00:05:32 [debug] cassandra_repl_strategy = "SimpleStrategy"
2020/04/11 00:05:32 [debug] cassandra_schema_consensus_timeout = 180000
2020/04/11 00:05:32 [debug] cassandra_ssl = false
2020/04/11 00:05:32 [debug] cassandra_ssl_verify = false
2020/04/11 00:05:32 [debug] cassandra_timeout = 5000
2020/04/11 00:05:32 [debug] cassandra_username = "suadmincass"
2020/04/11 00:05:32 [debug] client_body_buffer_size = "8k"
2020/04/11 00:05:32 [debug] client_header_buffer_size = "1k"
2020/04/11 00:05:32 [debug] client_max_body_size = "0"
2020/04/11 00:05:32 [debug] client_ssl = false
2020/04/11 00:05:32 [debug] database = "cassandra"
2020/04/11 00:05:32 [debug] db_cache_ttl = 0
2020/04/11 00:05:32 [debug] db_resurrect_ttl = 30
2020/04/11 00:05:32 [debug] db_update_frequency = 5
2020/04/11 00:05:32 [debug] db_update_propagation = 1
2020/04/11 00:05:32 [debug] dns_error_ttl = 1
2020/04/11 00:05:32 [debug] dns_hostsfile = "/etc/hosts"
2020/04/11 00:05:32 [debug] dns_no_sync = false
2020/04/11 00:05:32 [debug] dns_not_found_ttl = 30
2020/04/11 00:05:32 [debug] dns_order = {"LAST","SRV","A","CNAME","AAAA"}
2020/04/11 00:05:32 [debug] dns_resolver = {}
2020/04/11 00:05:32 [debug] dns_stale_ttl = 4
2020/04/11 00:05:32 [debug] error_default_type = "text/plain"
2020/04/11 00:05:32 [debug] headers = {"server_tokens","latency_tokens"}
2020/04/11 00:05:32 [debug] large_client_header_buffers_number = 4
2020/04/11 00:05:32 [debug] large_client_header_buffers_size = "8k"
2020/04/11 00:05:32 [debug] log_anonymize = "OFF"
2020/04/11 00:05:32 [debug] log_level = "notice"
2020/04/11 00:05:32 [debug] lru_cache_size = 500000
2020/04/11 00:05:32 [debug] lua_package_cpath = ""
2020/04/11 00:05:32 [debug] lua_package_path = "./?.lua;./?/init.lua;"
2020/04/11 00:05:32 [debug] lua_socket_pool_size = 30
2020/04/11 00:05:32 [debug] lua_ssl_verify_depth = 1
2020/04/11 00:05:32 [debug] maria_database = "kong"
2020/04/11 00:05:32 [debug] maria_host = "127.0.0.1"
2020/04/11 00:05:32 [debug] maria_password = "******"
2020/04/11 00:05:32 [debug] maria_port = 3306
2020/04/11 00:05:32 [debug] maria_socket_keepalive_timeout = 60000
2020/04/11 00:05:32 [debug] maria_socket_pool_size = 20
2020/04/11 00:05:32 [debug] maria_ssl = false
2020/04/11 00:05:32 [debug] maria_ssl_verify = false
2020/04/11 00:05:32 [debug] maria_timeout = 10000
2020/04/11 00:05:32 [debug] maria_user = "kong"
2020/04/11 00:05:32 [debug] mem_cache_size = "128m"
2020/04/11 00:05:32 [debug] nginx_admin_directives = {}
2020/04/11 00:05:32 [debug] nginx_daemon = "off"
2020/04/11 00:05:32 [debug] nginx_http_directives = {}
2020/04/11 00:05:32 [debug] nginx_optimizations = true
2020/04/11 00:05:32 [debug] nginx_proxy_directives = {}
2020/04/11 00:05:32 [debug] nginx_sproxy_directives = {}
2020/04/11 00:05:32 [debug] nginx_stream_directives = {}
2020/04/11 00:05:32 [debug] nginx_user = "nobody nobody"
2020/04/11 00:05:32 [debug] nginx_worker_processes = "auto"
2020/04/11 00:05:32 [debug] origins = {}
2020/04/11 00:05:32 [debug] pg_database = "kong"
2020/04/11 00:05:32 [debug] pg_host = "127.0.0.1"
2020/04/11 00:05:32 [debug] pg_port = 5432
2020/04/11 00:05:32 [debug] pg_ssl = false
2020/04/11 00:05:32 [debug] pg_ssl_verify = false
2020/04/11 00:05:32 [debug] pg_timeout = 5000
2020/04/11 00:05:32 [debug] pg_user = "kong"
2020/04/11 00:05:32 [debug] plugins = {"bundled"}
2020/04/11 00:05:32 [debug] prefix = "/usr/local/kong/"
2020/04/11 00:05:32 [debug] proxy_access_log = "logs/access.log"
2020/04/11 00:05:32 [debug] proxy_error_log = "logs/error.log"
2020/04/11 00:05:32 [debug] proxy_listen = {"0.0.0.0:8000","0.0.0.0:8443 ssl"}
2020/04/11 00:05:32 [debug] real_ip_header = "X-Real-IP"
2020/04/11 00:05:32 [debug] real_ip_recursive = "off"
2020/04/11 00:05:32 [debug] ssl_cipher_suite = "modern"
2020/04/11 00:05:32 [debug] ssl_ciphers = "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256"
2020/04/11 00:05:32 [debug] stream_listen = {"off"}
2020/04/11 00:05:32 [debug] trusted_ips = {}
2020/04/11 00:05:32 [debug] upstream_keepalive = 60
2020/04/11 00:05:32 [debug] validate_certs_refresh_interval = 300
2020/04/11 00:05:32 [debug] validate_certs_warn_before = 7
2020/04/11 00:05:32 [verbose] prefix in use: /usr/local/kong
2020/04/11 00:05:32 [debug] resolved Cassandra contact point '2001:283:4000:2001::5167' to: [2001:283:4000:2001::5167]
*IPv6_1 <<< my custom log*
Error:
./kong/cmd/migrations.lua:107: [Cassandra error] all hosts tried for query failed. [2001:283:4000:2001::5167]: host seems unhealthy, considering it down (host or service not provided, or not known)
stack traceback:
        [C]: in function 'assert'
        ./kong/cmd/migrations.lua:107: in function 'cmd_exec'
        ./kong/cmd/init.lua:90: in function <./kong/cmd/init.lua:90>
        [C]: in function 'xpcall'
        ./kong/cmd/init.lua:90: in function <./kong/cmd/init.lua:47>
        /etc/kong/openresty_temp_script.lua:6: in function 'file_gen'
        init_worker_by_lua:49: in function <init_worker_by_lua:47>
        [C]: in function 'xpcall'
        init_worker_by_lua:56: in function <init_worker_by_lua:54>
bash-4.2$

smuthunoori · April 11, 2020, 1:28am

Are you using nginx as well? If yes, then did you expose your env variables through nginx conf file?

angrosz · April 11, 2020, 5:05pm

I have Kong installed in k8s. I’m NOT configuring Kong through conf file, only through env variables. What nginx conf file do you mean?

angrosz · April 11, 2020, 6:16pm

I did the following modification in socket.lua (look for ADDRESS_FAMILY)

bash-4.2# cat /usr/local/share/lua/5.1/cassandra/socket.lua
local type = type

----------------------------
-- LuaSocket proxy metatable
----------------------------

local proxy_mt

do
  local tostring = tostring
  local concat = table.concat
  local pairs = pairs

  local function flatten(v, buf)
    if type(v) == 'string' then
      buf[#buf+1] = v
    elseif type(v) == 'table' then
      for i = 1, #v do
        flatten(v[i], buf)
      end
    end
  end

  proxy_mt = {
    send = function(self, data)
      if type(data) == 'table' then
        local buffer = {}
        flatten(data, buffer)
        data = concat(buffer)
      end

      return self.sock:send(data)
    end,
    getreusedtimes = function() return 0 end,
    settimeout = function(self, t)
      if t then
        t = t/1000
      end
      self.sock:settimeout(t)
    end,
    setkeepalive = function(self)
      self.sock:close()
      return true
    end,
    close = function(self)
      -- LuaSec dismisses the return value from sock:close(), so we override
      -- sock:close() here to ensure that we always return non-nil from it,
      -- even when wrapped by LuaSec
      self.sock:close()
      return 1
    end,
    sslhandshake = function(self, reused_session, _, verify, opts)
      opts = opts or {}
      local return_bool = reused_session == false

      local ssl = require 'ssl'
      local params = {
        mode = 'client',
        protocol = 'tlsv1_2',
        key = opts.key,
        certificate = opts.cert,
        cafile = opts.cafile,
        verify = verify and 'peer' or 'none',
        options = 'all'
      }

      local sock, err = ssl.wrap(self.sock, params)
      if not sock then
        return return_bool and false or nil, err
      end

      local ok, err = sock:dohandshake()
      if not ok then
        return return_bool and false or nil, err
      end

      -- purge memoized closures
      for k, v in pairs(self) do
        if type(v) == 'function' then
          self[k] = nil
        end
      end

      self.sock = sock

      return return_bool and true or self
    end
  }

  proxy_mt.__tostring = function(self)
    return tostring(self.sock)
  end

  proxy_mt.__index = function(self, key)
    local override = proxy_mt[key]
    if override then
      return override
    end

    local orig = self.sock[key]
    if type(orig) == 'function' then
      local f = function(_, ...)
        return orig(self.sock, ...)
      end
      self[key] = f
      return f
    elseif orig then
      return orig
    end
  end
end

---------
-- Module
---------

local _M = {
  luasocket_mt = proxy_mt,
  _VERSION = '1.0.0'
}

-----------------------
-- ngx_lua/plain compat
-----------------------

local COSOCKET_PHASES = {
  rewrite = true,
  access = true,
  content = true,
  timer = true,
  ssl_cert = true,
  ssl_session_fetch = true
}

local forced_luasocket_phases = {}
local forbidden_luasocket_phases = {}

do
  local setmetatable = setmetatable

  if ngx then
    local log, WARN, INFO = ngx.log, ngx.WARN, ngx.INFO
    local get_phase = ngx.get_phase
    local ngx_socket = ngx.socket

    function _M.tcp(...)
      local phase = get_phase()
      if not forced_luasocket_phases[phase]
         and COSOCKET_PHASES[phase]
         or forbidden_luasocket_phases[phase] then
        return ngx_socket.tcp(...)
      end

      -- LuaSocket
      if phase ~= 'init' then
        if forced_luasocket_phases[phase] then
          log(INFO, 'support for cosocket in this context, but LuaSocket forced')
        else
          log(WARN, 'no support for cosockets in this context, falling back to LuaSocket')
        end
      end

      local socket = require 'socket'

      if os.getenv("ADDRESS_FAMILY") == "ipv6" then
        print("IPv6_1")
        return setmetatable({
          sock = socket.tcp6(...)
        }, proxy_mt)
      else
        return setmetatable({
          sock = socket.tcp(...)
        }, proxy_mt)
      end
    end
  else
    local socket = require 'socket'

    function _M.tcp(...)
      if os.getenv("ADDRESS_FAMILY") == "ipv6" then
        print("IPv6_2")
        return setmetatable({
          sock = socket.tcp6(...)
        }, proxy_mt)
      else
        return setmetatable({
          sock = socket.tcp(...)
        }, proxy_mt)
      end
    end
  end
end

---------------------------------------
-- Disabling/forcing LuaSocket fallback
---------------------------------------

do
  local function check_phase(phase)
    if type(phase) ~= 'string' then
      local info = debug.getinfo(2)
      local err = string.format("bad argument #1 to '%s' (%s expected, got %s)",
                                info.name, 'string', type(phase))
      error(err, 3)
    end
  end

  function _M.force_luasocket(phase, force)
    check_phase(phase)
    forced_luasocket_phases[phase] = force
  end

  function _M.disable_luasocket(phase, disable)
    check_phase(phase)
    forbidden_luasocket_phases[phase] = disable
  end
end

return _M

Look at my second scenario. My custom log “IPv6_1” has been displayed.

smuthunoori · April 12, 2020, 5:36am

Yes, in your second scenario, it is using tcp6() and hence was able to resolve the address provided. However, even in scenario 1, the IPv6_1 message should be displayed as well. Can you try printing the value of “os.getenv(“ADDRESS_FAMILY”)” in case of scenario 1. I suspect if the program is able to see that env variable or not.

In my case, I am using both kong.conf and kong_nginx.conf files. Hence, I had to expose that env variable through kong_nginx.conf file as well.

angrosz · April 12, 2020, 8:21am

Can you please share your changes made in kong.conf and kong_nginx.conf?
What are values of dns_order and dns_resolver when you are executing kong migrations list?

Can you share the whole files kong.conf and kong_nginx.conf?

smuthunoori · April 14, 2020, 4:17am

Here are some specific settings in my kong.conf and kong_nginx.conf

kong.conf

dns_order = LAST,SRV,A,CNAME,AAAA
assandra_contact_points = cassandra.default.svc.cluster.local

kong_nginx.conf:

env ADDRESS_FAMILY;
http {
resolver [2001:10::1];
init_by_lua_block {
require ‘resty.core’
Kong = require ‘kong’
Kong.init()
}

init_worker_by_lua_block {
Kong.init_worker()
}
…
}

angrosz · April 20, 2020, 10:10pm

To solve my SECOND SCENARIO I had to remove square brackets from IPv6 address in /usr/local/share/lua/5.1/kong/db/strategies/cassandra/connector.lua:

    local dns_tools = require "kong.tools.dns"

    kong_config.dns_no_sync = true

    local dns = dns_tools(kong_config)

    for i, cp in ipairs(kong_config.cassandra_contact_points) do
      local ip, err = dns.toip(cp)
      if not ip then
        log.error("could not resolve Cassandra contact point '%s': %s", cp, err)
      else
        ip = string.gsub(ip, "%[*%]*", "")       <------------------- added this line
        log.debug("resolved Cassandra contact point '%s' to: %s", cp, ip)
        resolved_contact_points[i] = ip
      end
    end

I’m still NOT able to solve my FIRST SCENARIO

angrosz · April 21, 2020, 3:58pm

Hey Srinivas,

Could you please share what changes you made in /usr/local/share/lua/5.1/socket/http.lua?

Best Regards,
Ania

Topic		Replies	Views
KONG cassandra error	1	479	February 8, 2019
[Kong 1.1.1] [k8s][IPv6][Cassandra db] How to configure Kong 1.1.1 to contact Cassandra via IPv6 Installation/Setup	0	563	April 5, 2020
Kong V14 - Kong error Installation/Setup	8	1520	August 31, 2018
Kong 0.14.1 backed by Datastax Enterprise Cassandra 6.0.2 fails to start Installation/Setup	1	673	October 10, 2018
Issue with Kong backed by Cassandra	3	647	August 18, 2021

Run Kong as Kubernetes Service with IPv6 Support

Related Topics