Kong declarative config upstream healthcheck fails

Hi, having an issue with upstream health being set to “health”:“HEALTHCHECKS_OFF”, and not sure why, if i actually try to send request through it seems to work, but im not sure why HEALTHCHECKS_OFF is returned in query.

  - name: my_service
    host: my_service_host
    - name: my_service
      - my_service_host
      strip_path: false

  - name: my_service.upstream
    hash_fallback: none
    hash_on: header
    hash_on_header: X-User-ID
    - target: my_service.route53.srv
      weight: 100

/ # curl -X GET http://localhost:8001/upstreams/my_service.upstream/health

when i send the curl to cehck the health, i see this error in the kong logs.

2019/08/06 11:56:51 [error] 292#0: *394962 [lua] targets.lua:240: page_collection(): failed getting upstream health: balancer not found, client:, server: kong_admin, request: "GET /upstreams/my_service.upstream/health HTTP/1.1", host: "localhost:8001"

and none of the targets get listed in targets, there should be about 20 targets, in kong 0.14 there is a different issue with DNS_ERROR, but it actually lists all the targets.

/ # curl -X GET http://localhost:8001/upstreams/my_service.upstream/targets/all

Is there any issues with the declarative configuration used to create the service/route/upstream targets?

after kong reload or start, get this error

2019/08/06 13:45:40 [error] 584#0: *434182 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/resty/dns/balancer/base.lua:798: expected a hostname (string), got nil

Hi, mnk0.
Your configuration is missing the target port, so Kong can’t use it. It should be something like:

    - target: my_service.route53.srv:80

Also, this configuration does not define any health check, that’s why Kong is reporting HEALTHCHECKS_OFF. Here is an example of a passive health check:

  - name: my_service.upstream
          - 200
          - 304
          successes: 1
        type: http
          http_failures: 5
          - 429
          - 500
          - 503
          tcp_failures: 2
          timeouts: 2

You can read more about health checks on https://docs.konghq.com/1.2.x/health-checks-circuit-breakers/

Thanks for the reply!! been blocked on this for several days,

i tried some variations of adding the healthchecks and still failed, but if i dont add them explicityly in the config.yaml my understanding was default values would be used? testing this out now.

2- also, the target upstream is AWS ECS service discovery endpoint. so shouldnt kong pick up the port automatically here? my understanding is that it should retrieve the port from the lua-resty-dns-client. although i 've seen some issues related to this – for example : https://github.com/Kong/kong/issues/4781

for example the port is returned from dig query.

/etc/kong # dig staging-falcon-reader.awsvpc-private ANY

; <<>> DiG 9.14.3 <<>> staging-falcon-reader.awsvpc-private ANY
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7095
;; flags: qr rd ra; QUERY: 1, ANSWER: 8, AUTHORITY: 4, ADDITIONAL: 8

;staging-falcon-reader.awsvpc-private. IN ANY

staging-falcon-reader.awsvpc-private. 3 IN SRV 1 1 35946 ccf0e092-5ac8-4c60-b9cd-2a310988ae53.staging-falcon-reader.awsvpc-private.
staging-falcon-reader.awsvpc-private. 3 IN SRV 1 1 35804 934e7d8a-8d66-49b3-ae1d-a660991a3a55.staging-falcon-reader.awsvpc-private.
staging-falcon-reader.awsvpc-private. 3 IN SRV 1 1 35453 dcebbf0c-48bb-44db-a333-432360e2b7b3.staging-falcon-reader.awsvpc-private.
staging-falcon-reader.awsvpc-private. 3 IN SRV 1 1 35679 16ab2c0b-18d6-4011-b47f-3c18d239244c.staging-falcon-reader.awsvpc-private.
staging-falcon-reader.awsvpc-private. 3 IN SRV 1 1 32867 0bef3aa7-3b68-4032-98ed-089e72578edc.staging-falcon-reader.awsvpc-private.
staging-falcon-reader.awsvpc-private. 3 IN SRV 1 1 35894 b7807815-f982-4cca-b43d-a6f567323e98.staging-falcon-reader.awsvpc-private.
staging-falcon-reader.awsvpc-private. 3 IN SRV 1 1 35811 056f7f32-21f1-47a3-a3f3-f3258b4bb118.staging-falcon-reader.awsvpc-private.
staging-falcon-reader.awsvpc-private. 3 IN SRV 1 1 35760 ed0402f3-234f-4d9b-84e0-c043ba2db2bb.staging-falcon-reader.awsvpc-private.

additionally, tried the configuration with and without that additional block of defining the health check parameters, but still same result.

/etc/kong # curl -X GET http://localhost:8001/upstreams/falcon.upstream

/etc/kong # curl -X GET http://localhost:8001/upstreams/falcon.upstream/health

is there any configuration needed for changing the DNS acceptable payload size? or way to verify that the lua-resty-dns client can actually resolve the srv dns records?

© 2018 Kong Inc.    Terms  •  Privacy  •  FAQ