Consumer value is empty in Prometheus kong_http_requests_total

I am using Kong 3.1. I have set per_consumer flag to true in my Prometheus global plugin but still in Prometheus metrics logs I cannot see consumer value.
For some services log, it is shown with correct consumer value but not for all services’ kong_http_requests_total log. Example below:


@Sachin_Ghumbre Do you have authentication plugin set on service/route level?

No @trustworthygoblin . I am using separate OAuth token endpoint with Global OAuth plugin (client credentials grant type) to generate the token. Generated token will be passed in authorization header while making a call to other APIs. Here, sometimes consumer value is captured in the prometheus log and sometimes not.

Interestingly, I can see all 4xx/5xx API calls do not have consumer value in prometheus log kong_http_requests_total . Also, my separate Token Endpoint (OAuth) and couple of other APIs with 2xx are also not showing consumer value. Rest all successful APIs show correct consumer value.