Kong fails to create balancer for upstream

Questions
1

Hi all,

I’m running into a couple of issues when using Kong 1.3.0-rc1 databaseless with the Kubernetes Ingress controller.

#1: This health check issue seems to be present all the time, though it gets worse as more routes are deployed.

2019/08/09 18:34:12 [error] 160#0: *147587 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/runloop/balancer.lua:249: attempt to index local 'healthchecker' (a nil value)
stack traceback:
coroutine 0:
        /usr/local/share/lua/5.1/kong/runloop/balancer.lua: in function 'callback'
        /usr/local/share/lua/5.1/resty/dns/balancer/base.lua:1270: in function </usr/local/share/lua/5.1/resty/dns/balancer/base.lua:1269>, context: ngx.timer
2019/08/09 18:34:42 [crit] 160#0: *101490 [lua] balancer.lua:640: on_upstream_event(): failed creating balancer for cloudproxy.arnavdatabus.svc: timeout waiting for balancer for 00aca459-9400-5dc9-905d-1908b6b36e34, context: ngx.timer
2019/08/09 18:34:42 [error] 160#0: *101490 [lua] balancer.lua:609: on_target_event(): target create: balancer not found for cloudproxy.arnavdatabus.svc, context: ngx.timer
2019/08/09 18:34:42 [error] 160#0: *101490 [lua] events.lua:194: do_handlerlist(): worker-events: event callback failed; source=lua-resty-healthcheck [service.namespace.svc], event=clear, pid=164 error='/usr/local/share/lua/5.1/resty/healthcheck.lua:225: attempt to index field 'targets' (a nil value)
stack traceback:
        /usr/local/share/lua/5.1/resty/healthcheck.lua:225: in function 'get_target'
        /usr/local/share/lua/5.1/resty/healthcheck.lua:942: in function </usr/local/share/lua/5.1/resty/healthcheck.lua:940>
        [C]: in function 'xpcall'
        /usr/local/share/lua/5.1/resty/worker/events.lua:185: in function 'do_handlerlist'
        /usr/local/share/lua/5.1/resty/worker/events.lua:217: in function 'do_event_json'
        /usr/local/share/lua/5.1/resty/worker/events.lua:361: in function 'poll'
        /usr/local/share/lua/5.1/resty/healthcheck.lua:1326: in function 'new'
        /usr/local/share/lua/5.1/kong/runloop/balancer.lua:339: in function 'create_healthchecker'
        /usr/local/share/lua/5.1/kong/runloop/balancer.lua:423: in function 'create_balancer'
        /usr/local/share/lua/5.1/kong/runloop/balancer.lua:760: in function 'init'
        /usr/local/share/lua/5.1/kong/runloop/handler.lua:812: in function </usr/local/share/lua/5.1/kong/runloop/handler.lua:811>', data={}, context: ngx.timer

#2: The official prometheus plugin does not appear to be working. When querying the endpoint, I only receive 500 internal server errors.
/plugins endpoint

{
  "created_at": 1565376474,
  "config": {},
  "id": "86e6f637-64e2-5672-920c-32e17e76e0fe",
  "service": null,
  "enabled": true,
  "tags": null,
  "consumer": null,
  "run_on": "first",
  "name": "prometheus",
  "route": null,
  "protocols": [
    "grpc",
    "grpcs",
    "http",
    "https"
  ]
},

/metrics endpoint:

{
  "message": "An unexpected error occurred"
}

Kong/logs/error.log

2019/08/09 18:51:07 [error] 162#0: *174040 [kong] exporter.lua:88 prometheus: plugin is not initialized, please make sure  'prometheus_metrics' shared dict is present in nginx template, client: 10.0.85.66, server: kong_prometheus_exporter, request: "GET /metrics HTTP/1.1", host: "10.0.48.81:9542"
2019/08/09 18:51:07 [error] 162#0: *174040 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/pdk/private/phases.lua:66: no phase in kong.ctx.core.phase
stack traceback:
coroutine 0:
        [C]: in function 'error'
        /usr/local/share/lua/5.1/kong/pdk/private/phases.lua:66: in function 'check_phase'
        /usr/local/share/lua/5.1/kong/pdk/response.lua:662: in function 'collect'
        content_by_lua(servers.conf:12):3: in main chunk, client: ..., server: kong_prometheus_exporter, request: "GET /metrics HTTP/1.1", host: "...:9542"

#3: Kong fails to create/find upstreams for upstreams. I think this is related to #1, but I’m listing it separately just in case.

kong/logs/error.log

2019/08/09 18:50:44 [error] 193#0: *173385 [lua] balancer.lua:634: on_upstream_event(): upstream not found for 00aca459-9400-5dc9-905d-1908b6b36e34, context: ngx.timer
2019/08/09 18:50:44 [error] 193#0: *173385 [lua] balancer.lua:634: on_upstream_event(): upstream not found for 26c43ca3-80ff-5862-98d5-e63d327ff612, context: ngx.timer
2019/08/09 18:50:44 [error] 193#0: *173385 [lua] balancer.lua:634: on_upstream_event(): upstream not found for be153462-f6dd-5574-8644-80a980f68141, context: ngx.timer
2019/08/09 18:50:44 [error] 193#0: *173385 [lua] balancer.lua:603: on_target_event(): target create: upstream not found for be153462-f6dd-5574-8644-80a980f68141, context: ngx.timer
2019/08/09 18:50:44 [error] 193#0: *173385 [lua] balancer.lua:603: on_target_event(): target create: upstream not found for 26c43ca3-80ff-5862-98d5-e63d327ff612, context: ngx.timer
2019/08/09 18:50:44 [error] 193#0: *173385 [lua] balancer.lua:603: on_target_event(): target create: upstream not found for 00aca459-9400-5dc9-905d-1908b6b36e34, context: ngx.timer

In addition I have noticed that after some time, Kong ‘looses’ all of its configuration data. (making me think that Kong itself has crashed). In these cases I notice errors like this:

kong/logs/error.log

019/08/09 18:50:45 [notice] 1#0: signal 17 (SIGCHLD) received from 161
2019/08/09 18:50:45 [alert] 1#0: worker process 161 exited on signal 9
2019/08/09 18:50:45 [notice] 1#0: start worker process 265
2019/08/09 18:50:45 [notice] 1#0: signal 29 (SIGIO) received

/upstreams

{
  "next": null,
  "data": []
}

/routes

{
  "next": null,
  "data": []
}

/plugins

{
  "next": null,
  "data": []
}

Any help would be appreciated,

Thanks,
Arnav

2

Thanks for the bug report. I’ll try to reproduce with what you’ve provided but if possible could I also get:

kic version
k8s version
kong.yml
how did you run kong (helm or yaml)

3

I tried to reproduce with some assumptions and am unable to

kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.8", GitCommit:"a89f8c11a5f4f132503edbc4918c98518fd504e3", GitTreeState:"clean", BuildDate:"2019-04-23T04:52:31Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-25T23:41:27Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

helm init --wait
helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/
helm repo update
helm install stable/kong --set ingressController.enabled=true,postgresql.enabled=false,env.database=off,image.tag=1.3rc1

curl https://raw.githubusercontent.com/Kong/kubernetes-ingress-controller/master/deploy/manifests/dummy-application.yaml | kubectl create -f -
export HOST=$(kubectl get nodes --namespace default -o jsonpath='{.items[0].status.addresses[0].address}')
export ADMIN_PORT=$(kubectl get svc --namespace default famous-sheep-kong-admin -o jsonpath='{.spec.ports[0].nodePort}')
export PROXY_PORT=$(kubectl get svc --namespace default famous-sheep-kong-proxy -o jsonpath='{.spec.ports[0].nodePort}')
export PROXY=$HOST:$PROXY_PORT

curl -sL bit.ly/echo-server | kubectl apply -f -

echo "
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: demo
spec:
  rules:
  - http:
      paths:
      - path: /foo
        backend:
          serviceName: echo
          servicePort: 80
" | kubectl apply -f -

curl -i $PROXY/foo
4

Versions:
KIC: 0.5.0
K8s:

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.8-eks-a977ba", GitCommit:"a977bab148535ec195f12edc8720913c7b943f9c", GitTreeState:"clean", BuildDate:"2019-07-29T20:47:04Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

kong.yaml:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: kongplugins.configuration.konghq.com
spec:
  group: configuration.konghq.com
  version: v1
  scope: Namespaced
  names:
    kind: KongPlugin
    plural: kongplugins
    shortNames:
    - kp
  additionalPrinterColumns:
  - name: Plugin-Type
    type: string
    description: Name of the plugin
    JSONPath: .plugin
  - name: Age
    type: date
    description: Age
    JSONPath: .metadata.creationTimestamp
  - name: Disabled
    type: boolean
    description: Indicates if the plugin is disabled
    JSONPath: .disabled
    priority: 1
  - name: Config
    type: string
    description: Configuration of the plugin
    JSONPath: .config
    priority: 1
  validation:
    openAPIV3Schema:
      required:
      - plugin
      properties:
        plugin:
          type: string
        disabled:
          type: boolean
        config:
          type: object
        run_on:
          type: string
          enum:
          - first
          - second
          - all
        protocols:
          type: array
          items:
            type: string
            enum:
            - http
            - https
            - tcp
            - tls


---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: kongconsumers.configuration.konghq.com
spec:
  group: configuration.konghq.com
  version: v1
  scope: Namespaced
  names:
    kind: KongConsumer
    plural: kongconsumers
    shortNames:
    - kc
  additionalPrinterColumns:
  - name: Username
    type: string
    description: Username of a Kong Consumer
    JSONPath: .username
  - name: Age
    type: date
    description: Age
    JSONPath: .metadata.creationTimestamp
  validation:
    openAPIV3Schema:
      properties:
        username:
          type: string
        custom_id:
          type: string

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: kongcredentials.configuration.konghq.com
spec:
  group: configuration.konghq.com
  version: v1
  scope: Namespaced
  names:
    kind: KongCredential
    plural: kongcredentials
  additionalPrinterColumns:
  - name: Credential-type
    type: string
    description: Type of credential
    JSONPath: .type
  - name: Age
    type: date
    description: Age
    JSONPath: .metadata.creationTimestamp
  - name: Consumer-Ref
    type: string
    description: Owner of the credential
    JSONPath: .consumerRef
  validation:
    openAPIV3Schema:
      required:
      - consumerRef
      - type
      properties:
        consumerRef:
          type: string
        type:
          type: string

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: kongingresses.configuration.konghq.com
spec:
  group: configuration.konghq.com
  version: v1
  scope: Namespaced
  names:
    kind: KongIngress
    plural: kongingresses
    shortNames:
    - ki
  validation:
    openAPIV3Schema:
      properties:
        upstream:
          type: object
        route:
          properties:
            methods:
              type: array
              items:
                type: string
            regex_priority:
              type: integer
            strip_path:
              type: boolean
            preserve_host:
              type: boolean
            protocols:
              type: array
              items:
                type: string
                enum:
                - http
                - https
        proxy:
          type: object
          properties:
            protocol:
              type: string
              enum:
              - http
              - https
            path:
              type: string
              pattern: ^/.*$
            retries:
              type: integer
              minimum: 0
            connect_timeout:
              type: integer
              minimum: 0
            read_timeout:
              type: integer
              minimum: 0
            write_timeout:
              type: integer
              minimum: 0
        upstream:
          type: object
          properties:
            hash_on:
              type: string
            hash_on_cookie:
              type: string
            hash_on_cookie_path:
              type: string
            hash_on_header:
              type: string
            hash_fallback_header:
              type: string
            hash_fallback:
              type: string
            slots:
              type: integer
              minimum: 10
            healthchecks:
              type: object
              properties:
                active:
                  type: object
                  properties:
                    concurrency:
                      type: integer
                      minimum: 1
                    timeout:
                      type: integer
                      minimum: 0
                    http_path:
                      type: string
                      pattern: ^/.*$
                    healthy: &healthy
                      type: object
                      properties:
                        http_statuses:
                          type: array
                          items:
                            type: integer
                        interval:
                          type: integer
                          minimum: 0
                        successes:
                          type: integer
                          minimum: 0
                    unhealthy: &unhealthy
                      type: object
                      properties:
                        http_failures:
                          type: integer
                          minimum: 0
                        http_statuses:
                          type: array
                          items:
                            type: integer
                        interval:
                          type: integer
                          minimum: 0
                        tcp_failures:
                          type: integer
                          minimum: 0
                        timeout:
                          type: integer
                          minimum: 0
                passive:
                  type: object
                  properties:
                    healthy: *healthy
                    unhealthy: *unhealthy

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kong-serviceaccount
  namespace: namespace

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: kong-ingress-clusterrole
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - nodes
  - pods
  - secrets
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "extensions"
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - "extensions"
  resources:
  - ingresses/status
  verbs:
  - update
- apiGroups:
  - "configuration.konghq.com"
  resources:
  - kongplugins
  - kongcredentials
  - kongconsumers
  - kongingresses
  verbs:
  - get
  - list
  - watch

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: kong-ingress-role
  namespace: namespace
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - pods
  - secrets
  - namespaces
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - configmaps
  resourceNames:
  # Defaults to "<election-id>-<ingress-class>"
  # Here: "<ingress-controller-leader>-<kong>"
  # This has to be adapted if you change either parameter
  # when launching the kong-ingress-controller.
  - "ingress-controller-leader-kong"
  verbs:
  - get
  - update
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - create
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - get

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: kong-ingress-role-nisa-binding
  namespace: namespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kong-ingress-role
subjects:
- kind: ServiceAccount
  name: kong-serviceaccount
  namespace: namespace

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kong-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kong-ingress-clusterrole
subjects:
- kind: ServiceAccount
  name: kong-serviceaccount
  namespace: namespace

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: kong-server-blocks
  namespace: namespace
data:
  servers.conf: |
    # Prometheus metrics server
    server {
        server_name kong_prometheus_exporter;
        listen 0.0.0.0:9542; # can be any other port as well

        access_log off;
        location /metrics {
            default_type text/plain;
            content_by_lua_block {
                 local prometheus = require "kong.plugins.prometheus.exporter"
                 prometheus:collect()
            }
        }

        location /nginx_status {
            internal;
            access_log off;
            stub_status;
        }
    }
    # Health check server
    # TODO how to health check kong in dbless?
    server {
        server_name kong_health_check;
        listen 0.0.0.0:9001; # can be any other port as well

        access_log off;
        location /health {
          return 200;
        }
    }

---

apiVersion: extensions/v1beta1
kind: "Deployment"
metadata:
  labels:
    app: "kong"
  name: "-frontend----kong"
  namespace: "namespace"
spec:
  replicas: 1
  selector:
    matchLabels:
      role: frontend----kong
  strategy:
    rollingUpdate:
      maxSurge: 3
      maxUnavailable: 0
    type: "RollingUpdate"
  template:
    metadata:
      annotations:
        prometheus.io/port: "9542"
        prometheus.io/scrape: "true"
      labels:
        role: "frontend----kong"
        app: apimanagement
    spec:
      serviceAccountName: "kong-serviceaccount"
      initContainers:
      - name: init-logging
        image: busybox:1.30.1
        command: ['sh', '-c', 'mkdir -p /shared/logs/; chmod -R 777 /shared/logs/']
        volumeMounts:
        - name: temporal
          mountPath: "/shared"
      containers:
      - name: "-kong"
        image: "kong:1.3.0-rc1"
        imagePullPolicy: Always
        resources:
          requests:
            memory: "500Mi"
            cpu: "200m"
          limits:
            memory: "500Mi"
            cpu: "500m"
        env:
        - name: KONG_DATABASE
          value: "off"
        - name: KONG_NGINX_HTTP_INCLUDE
          value: "/kong/servers.conf"
        - name: KONG_ADMIN_ACCESS_LOG
          value: /dev/stdout
        - name: KONG_ADMIN_ERROR_LOG
          value: /dev/stderr
        - name: KONG_ADMIN_LISTEN
          value: 127.0.0.1:8444 ssl
        - name: KONG_LUA_PACKAGE_PATH
          value: "/usr/local/share/lua/5.1/?.lua;;"
        - name: KONG_CUSTOM_PLUGINS
          value: cloudlink-auth
        ports:
        - name: proxy
          containerPort: 8000
          protocol: TCP
        - name: proxy-ssl
          containerPort: 8443
          protocol: TCP
        - name: metrics
          containerPort: 9542
          protocol: TCP
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 9001
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 9001
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        volumeMounts:
        - name: kong-server-blocks
          mountPath: /kong
        - name: temporal
          mountPath: "/shared"
      - name: ingress-controller
        args:
        - /kong-ingress-controller
        # the kong URL points to the kong admin api server
        - --kong-url=https://localhost:8444
        - --admin-tls-skip-verify
        # Service from were we extract the IP address/es to use in Ingress status
        - --publish-service=namespace/kong-external
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: "kong-ingress-controller:0.5.0"
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
      volumes:
      - name: kong-server-blocks
        configMap:
          name: kong-server-blocks
      - name: temporal
        hostPath:
          path: "/shared"
---

apiVersion: v1
kind: Service
metadata:
  name: kong-external
  namespace: namespace
spec:
  type: LoadBalancer
  ports:
  - name: kong-proxy
    port: 80
    targetPort: 8000
    protocol: TCP
  - name: kong-proxy-ssl
    port: 443
    targetPort: 8443
    protocol: TCP
    protocol: TCP
  selector:
    role: frontend----kong
---
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: kong-prometheus
  namespace: namespace
  labels:
    global: "true"
plugin: prometheus

I am using yaml, not helm.

Let me know if there is anything else you need.

Thanks,
Arnav

5

Hi, did you manage to resolve the “attempt to index local ‘healthchecker’ (a nil value)” error? I am seeing the same error in my deployment, (Kong 1.4 + kong-ingress-controller 0.6.1 + Postgres DB).

Error message specifically is:

lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/runloop/balancer.lua:241: attempt to index local 'healthchecker' (a nil value)

Thanks!