Hava something wrong with the response of localhost:8001/metrics

Questions
1

Hello,
I followed the official Doc of the Prometheus
and I tried http://localhost:8001/metrics but have something wrong with a response. I’m not sure if it’s a Prometheus plugin issue or something I’m doing wrong. Thank you for reading

Kong Version 1.0.0rc2

localhost:8001/plugins

{ next: null, data: [ { created_at: 1544434246, config: { }, id: "020666d3-0f39-4fc2-80e6-6b8a9534d1c4", service: { id: "de838225-be9c-4e5c-99e1-f21b47970e03" }, name: "prometheus", api: null, consumer: null, route: null, enabled: true }, { created_at: 1544497752, config: { }, id: "302d6c16-e4ba-4574-b589-c27ed3bb6829", service: null, name: "prometheus", api: null, consumer: null, route: null, enabled: true } ] }

localhost:8001/metrics

# HELP kong_datastore_reachable Datastore reachable from Kong, 0 is unreachable
# TYPE kong_datastore_reachable gauge
kong_datastore_reachable 1
# HELP kong_nginx_http_current_connections Number of HTTP connections
# TYPE kong_nginx_http_current_connections gauge
kong_nginx_http_current_connections{state="accepted"} 19
kong_nginx_http_current_connections{state="active"} 1
kong_nginx_http_current_connections{state="handled"} 19
kong_nginx_http_current_connections{state="reading"} 0
kong_nginx_http_current_connections{state="total"} 131
kong_nginx_http_current_connections{state="waiting"} 0
kong_nginx_http_current_connections{state="writing"} 1
# HELP kong_nginx_metric_errors_total Number of nginx-lua-prometheus errors
# TYPE kong_nginx_metric_errors_total counter
kong_nginx_metric_errors_total 0
{"message":"An unexpected error occurred"}

error.log of Kong


2018/12/11 11:12:03 [error] 2303#0: *8748 [lua] init.lua:143: handle_error(): /usr/local/share/lua/5.1/lapis/application.lua:397: /usr/local/share/lua/5.1/kong/plugins/prometheus/api.lua:20: headers have already been sent
stack traceback:
        [C]: in function 'error'
        /usr/local/share/lua/5.1/kong/pdk/response.lua:553: in function 'collect'
        /usr/local/share/lua/5.1/kong/plugins/prometheus/api.lua:20: in function 

stack traceback:
        [C]: in function 'error'
        /usr/local/share/lua/5.1/lapis/application.lua:397: in function 'handler'
        /usr/local/share/lua/5.1/lapis/application.lua:130: in function 'resolve'
        /usr/local/share/lua/5.1/lapis/application.lua:161: in function 
        [C]: in function 'xpcall'
        /usr/local/share/lua/5.1/lapis/application.lua:159: in function 'dispatch'
        /usr/local/share/lua/5.1/lapis/nginx.lua:215: in function 'serve_admin_api'
        content_by_lua(nginx-kong.conf:175):2: in function , client: 223.104.5.223, server: kong_admin, request: "GET /metrics HTTP/1.1", host: "115.159.34.56:8001"
2018/12/11 11:12:03 [error] 2303#0: *8748 attempt to set ngx.status after sending out response headers, client: 223.104.5.223, server: kong_admin, request: "GET /metrics HTTP/1.1", host: "115.159.34.56:8001"
2018/12/11 11:12:03 [error] 2303#0: *8748 attempt to set ngx.header.HEADER after sending out response headers, client: 223.104.5.223, server: kong_admin, request: "GET /metrics HTTP/1.1", host: "115.159.34.56:8001"
2018/12/11 11:12:03 [error] 2303#0: *8748 attempt to set ngx.header.HEADER after sending out response headers, client: 223.104.5.223, server: kong_admin, request: "GET /metrics HTTP/1.1", host: "115.159.34.56:8001"
2018/12/11 11:12:03 [error] 2303#0: *8748 attempt to set ngx.header.HEADER after sending out response headers, client: 223.104.5.223, server: kong_admin, request: "GET /metrics HTTP/1.1", host: "115.159.34.56:8001"
2018/12/11 11:12:03 [error] 2303#0: *8748 attempt to set status 500 via ngx.exit after sending out the response status 200, client: 223.104.5.223, server: kong_admin, request: "GET /metrics HTTP/1.1",
1 Like
2

Hey, I hope you are fine!
I have the same issue, let’s wait for a reply.

3

This stacktrave seems to show that you are using a third-party Prometheus plugin for Kong. Is this the plugin you are using:

https://github.com/yciabaud/kong-plugin-
prometheus

If so, which version? The error appears to have been fixed in this commit:

You are free to use a third-party plugin, but note that Kong now bundled an official Prometheus plugin as well:

4

Hey @thibaultcha, thanks for your help!
I’m using the official release of Kong, but without success for now, my version of Kong is 0.14.1, plus I have one more problem on these metrics, Kong simply removes the plugin after 5, 10 minutes.

Thank you !

5

That is a separate issue, to which I answered in your other topic here. Let’s keep this thread on-topic :slight_smile:

So, I will reiterate my question: which Prometheus plugin are you using? Most likely the third-party one, which you should upgrade, or switch to the official Kong-maintained Prometheus plugin.

6

Hey , I’m not using a third-party Prometheus plugin for Kong, I am using bundled an official Prometheus plugin for Kong.
I haven’t found the version of the Prometheus plugin in the src. I have just found the version of the PrometheusHandler.version. 

PrometheusHandler.VERSION = “0.1.0”
7

Well, the api.lua file of the bundled Kong plugin does not seem to have ever had 20 lines of code…

8

:joy::joy: How should i do? The Kong version is 1.0.0rc2?

9

This was a bug in the Prometheus plugin in 1.0.0rc2, 1.0.0rc3 and 1.0.0rc4 which has been fixed with the following PR:

The fix will be available in subsequent releases of Kong.

10

@hbagdi The stacktrace point to line 20 of the api.lua file. The api.lua file of our Prometheus plugin never seems to have had 20 lines of code.

Additionally, the commit I pointed to in my previous message fixes this exact same error at line 20 of the api.lia file, I’m the community Prometheus plugin.

This doesn’t seem the be rested to the issues you pointed out.

11

This is pretty confusing.

I might be wrong here but I think 1.0.0rc2 is bundled with Prometheus plugin 0.3.x (x is 1 I think, we don’t pin it so I’m not sure, but based on dates of the tags, I think it’s 1), and 0.3.1 has a line 20.
But the other plugin also seems to be fixing the same issue at line 20 so this is indeed confusing.