Upstream with HEALTHCHECKS_OFF in one node

Hello everyone, hope somene here can help.

We have two kong nodes, they are using the same postgres database, so as I understand they forma kong cluster, correct ?

We are also using upstreams for our services, but only with active healthchecks for now. The configuration looks like this:

{
  "hash_on": "none",
  "healthchecks": {
    "passive": {
      "healthy": {
        "http_statuses": [
          200,
          201,
          202,
          203,
          204,
          205,
          206,
          207,
          208,
          226,
          300,
          301,
          302,
          303,
          304,
          305,
          306,
          307,
          308
        ],
        "successes": 0
      },
      "unhealthy": {
        "tcp_failures": 0,
        "timeouts": 0,
        "http_failures": 0,
        "http_statuses": [
          429,
          500,
          503
        ]
      }
    },
    "active": {
      "timeout": 5,
      "healthy": {
        "interval": 120,
        "successes": 3,
        "http_statuses": [
          200
        ]
      },
      "http_path": "/getversion/",
      "concurrency": 10,
      "unhealthy": {
        "timeouts": 3,
        "http_statuses": [
          404,
          429,
          500,
          501,
          502,
          503,
          504,
          505
        ],
        "http_failures": 3,
        "tcp_failures": 3,
        "interval": 10
      }
    }
  },
  "slots": 10000,
  "id": "183c79dc-1ff5-4ef9-895f-8c3a73a48ecb",
  "hash_on_cookie_path": "/",
  "created_at": 1544470425359,
  "name": "API",
  "hash_fallback": "none"
}

The configuration is exactly the same between the nodes, as it should be, since they are using the same database, right ?

When I request for the health of the upstream I get all targets as “HEALTHY” in one node but in the other one I get “HEALTHCHECKS_OFF” for all targets.

When I execute a request for this service through the node that has “HEALTHCHECKS_OFF” it works just fine.

Is there something we’re doing wrong ? I thought that all targets should be marked as “HEALTHY” in all nodes. We need this to configure our alerts correctly.

I am seeing the exact same thing.

We have 2 nodes on 0.14.1 and upstream changes are not properly propagated in the cluster.
If we remove an upstream on one node it still stays put on the other node.

Created an issue for it here: