Ingress controller fails to post config on IPv6 cluster

Questions
1

Hi, we are getting weird errors on an IPv6 cluster:

0901 07:28:59.683636 1 main.go:486] Running in Kubernetes Cluster version v1.16 (v1.16.3) - git (clean) commit b3cbbae08ec52a7fc73d334838e18d17e8512749 - platform linux/amd64
I0901 07:28:59.868832 1 main.go:189] kong version: 2.0.5
I0901 07:28:59.868856 1 main.go:198] Kong datastore: off
I0901 07:28:59.993200 1 controller.go:231] starting Ingress controller
I0901 07:28:59.993266 1 leaderelection.go:242] attempting to acquire leader lease default/kong-ingress-controller-leader-kong-kong…
E0901 07:28:59.993544 1 parser.go:1320] fetching KongPlugin ‘default/swagger-rewrite’: no KongPlugin or KongClusterPlugin was found
I0901 07:28:59.996337 1 status.go:210] new leader elected: kong-kong-855b7d4cf4-pqp5t
E0901 07:28:59.996984 1 reports.go:73] error initializing reports: lookup kong-hf.konghq.com on [fd74:ca9b:3a09:868c:172:18:0:800a]:53: server misbehaving
E0901 07:29:00.161812 1 controller.go:124] unexpected failure updating Kong configuration:
posting new config to /config: 400 Bad Request {“fields”:{“upstreams”:[{“targets”:[{“target”:“Invalid target; not a valid hostname or ip address”}]},{“targets”:[{“target”:“Invalid target; not a valid hostname or ip address”}]}]},“name”:“invalid declarative configuration”,“code”:14,“message”:“declarative config is invalid: {upstreams={{targets={{target=“Invalid target; not a valid hostname or ip address”}}},{targets={{target=“Invalid target; not a valid hostname or ip address”}}}}}”}
W0901 07:29:00.161837 1 queue.go:112] requeuing , err posting new config to /config: 400 Bad
Request {“fields”:{“upstreams”:[{“targets”:[{“target”:“Invalid target; not a valid hostname or ip address”}]},{“targets”:[{“target”:“Invalid target; not a valid hostname or ip address”}]}]},“name”:“invalid declarative configuration”,“code”:14,“message”:“declarative config is invalid: {upstreams={{targets={{target=“Invalid target; not a valid hostname or ip address”}}},{targets={{target=“Invalid target; not a valid hostname or ip address”}}}}}”}

Any ideas?

2

Can you share the Ingress resource you are trying to configure?
And then also please share the output of kubectl get endpoints for the service that is being proxied via Kong.

3

Ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: kong
    plugins.konghq.com: swagger-rewrite
  ...
    tenant: general
  name: eventsmanager
  namespace: default
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: eventsmanager
          servicePort: 80
        path: /rms/eventsmanager/v1(.*)
status:
  loadBalancer:
    ingress:
    - ip: 2001:db8::56

Rewrite plugin:

apiVersion: v1
items:
- apiVersion: configuration.konghq.com/v1
  config:
    replace:
      uri: /$(uri_captures[1])
  plugin: request-transformer
  kind: KongPlugin
  metadata:
    name: swagger-rewrite

Endpoint:

eventsmanager                             [fd74:ca9b:3a09:868c:172:18:0:4d59]:8080

ONe other thing I see is ingress controller is doing this requeueing thing for all services in the namespace. which fails after the initial configuration failures. Is this as expected?

W0901 16:41:09.524251       1 queue.go:112] requeuing default/kubernetes, err posting new config to /config: making HTTP request: Post "https://localhost:8444/config?check_hash=1": dial tcp [::1]:8444: connect: connection refused
E0901 16:41:12.857537       1 controller.go:124] unexpected failure updating Kong configuration:
posting new config to /config: making HTTP request: Post "https://localhost:8444/config?check_hash=1": dial tcp [::1]:8444: connect: connection refused
W0901 16:41:12.857594       1 queue.go:112] requeuing default/eventsmanager, err posting new config to /config: making HTTP request: Post "https://localhost:8444/config?check_hash=1": dial tcp [::1]:8444: connect: connection refused
4

Thanks a lot for being so responsive!

5

@hbagdi any thoughts?

6

@hbagdi can you please look at this?

7

Can you share the logs from the proxy container in the ingress-kong deployment?
It seems like Kong is not healthy.