Kong Prometheus always logged "failed getting upstream health: upstream not found"

Env: Kong 2.1.3; Prometheus Plugin: 0.9

We found lots of errors like the following when Prometheus crawls metrics:

2021/03/11 01:04:16 [error] 22#0: *3295802494 [kong] exporter.lua:208 failed getting upstream health: upstream not found, client: 1.1.1.1, server: kong_admin , request: “GET /metrics HTTP/1.1”, host: “apigw.cn:8001

we reviewed the Prometheus plugin code and found the logic:

local upstreams_dict = balancer.get_all_upstreams()
for key, upstream_id in pairs(upstreams_dict) do
local _, upstream_name = key:match("^([^:]*):(.-)$")
upstream_name = upstream_name and upstream_name or key
– based on logic from kong.db.dao.targets
local health_info
health_info, err = balancer.get_upstream_health(upstream_id)
if err then
kong.log.err("failed getting upstream health: ", err)
end

Then we accessed the Kong admin endpoints /cache/balancer:upstreams to get all upstreams and got each upstream by /cache/balancer:upstreams:{upstream_id}. Every upstream exits.

We got confused why these logs occurs. We’ll add some logs in Kong to identify which upstream is missing(will waiting for next deployment). Before that, Could anyone give us some clues? Thanks in advance.