Giter Club home page Giter Club logo

kong-plugin-prometheus's Issues

Load test the plugin

Load test the plugin to see if there are database / caching / performance issues

error when run with kong 0.13

When run this plugin in kong 0.13, report error below

2018/05/28 04:00:07 [debug] 45#0: *5121028 [lua] logger.lua:81: log(): Prometheus: logging metrics...
2018/05/28 04:00:07 [error] 45#0: *5121028 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/plugins/prometheus/logger.lua:87: bad argument #1 to 'string_gsub' (string expected, got nil)
stack traceback:
coroutine 0:
[C]: in function 'string_gsub'
/usr/local/share/lua/5.1/kong/plugins/prometheus/logger.lua:87: in function 'log'
.../local/share/lua/5.1/kong/plugins/prometheus/handler.lua:21: in function <.../local/share/lua/5.1/kong/plugins/prometheus/handler.lua:16>, context: ngx.timer, client: 192.168.141.0, server: 0.0.0.0:8000
2018/05/28 04:00:07 [info] 45#0: *5121026 client 192.168.141.0 closed keepalive connection

The latency metrics is incorrect

By default, nginx-lua-prometheus use a floating-point number for the elapsed time in seconds (including milliseconds as the decimal part). This place:

kong-plugin-prometheus/prometheus.lua

Lines 45 to 47 in 3e7e544

-- Default set of latency buckets, 5ms to 10s:
local DEFAULT_BUCKETS = {0.005, 0.01, 0.02, 0.03, 0.05, 0.075, 0.1, 0.2, 0.3,
0.4, 0.5, 0.75, 1, 1.5, 2, 3, 4, 5, 10}

But Kong use an interger precision in milliseconds. If we pass it to nginx-lua-prometheus directly, the latency accuracy is lost.

There came up with two solutions:

  1. Define custom buckets
  2. Divided the Kong latency by 1000

attempt to index global 'metrics' (a nil value)

I successfully added the plugin, however I seem to be getting this on /prometheus/metrics:

15/03/2018 10:11:112018/03/15 10:11:11 [error] 51#0: *3482 lua coroutine: runtime error: ...plugins/share/lua/5.1/kong/plugins/prometheus/logger.lua:21: attempt to index global 'metrics' (a nil value)
15/03/2018 10:11:11stack traceback:
15/03/2018 10:11:11coroutine 0:
15/03/2018 10:11:11	...plugins/share/lua/5.1/kong/plugins/prometheus/logger.lua: in function 'update_metric'
15/03/2018 10:11:11	...plugins/share/lua/5.1/kong/plugins/prometheus/logger.lua:144: in function 'logAdmin'
15/03/2018 10:11:11	...om-plugins/share/lua/5.1/kong/plugins/prometheus/api.lua:16: in function <...om-plugins/share/lua/5.1/kong/plugins/prometheus/api.lua:6>
15/03/2018 10:11:11coroutine 1:
15/03/2018 10:11:11	[C]: in function 'resume'
15/03/2018 10:11:11	/usr/local/share/lua/5.1/lapis/application.lua:393: in function 'handler'
15/03/2018 10:11:11	/usr/local/share/lua/5.1/lapis/application.lua:130: in function 'resolve'
15/03/2018 10:11:11	/usr/local/share/lua/5.1/lapis/application.lua:161: in function </usr/local/share/lua/5.1/lapis/application.lua:159>
15/03/2018 10:11:11	[C]: in function 'xpcall'
15/03/2018 10:11:11	/usr/local/share/lua/5.1/lapis/application.lua:159: in function 'dispatch'
15/03/2018 10:11:11	/usr/local/share/lua/5.1/lapis/nginx.lua:215: in function 'serve_admin_api'
15/03/2018 10:11:11	content_by_lua(nginx-kong.conf:150):2: in function <content_by_lua(nginx-kong.conf:150):1>, client: 10.42.150.236, server: kong_admin, request: "GET /prometheus/metrics HTTP/1.1", host: "api-admin.dev.mydomain.com"
15/03/2018 10:11:112018/03/15 10:11:11 [error] 51#0: *3482 [lua] init.lua:83: handle_error(): /usr/local/share/lua/5.1/lapis/application.lua:397: ...plugins/share/lua/5.1/kong/plugins/prometheus/logger.lua:21: attempt to index global 'metrics' (a nil value)
15/03/2018 10:11:11stack traceback:
15/03/2018 10:11:11	...plugins/share/lua/5.1/kong/plugins/prometheus/logger.lua: in function 'update_metric'
15/03/2018 10:11:11	...plugins/share/lua/5.1/kong/plugins/prometheus/logger.lua:144: in function 'logAdmin'
15/03/2018 10:11:11	...om-plugins/share/lua/5.1/kong/plugins/prometheus/api.lua:16: in function <...om-plugins/share/lua/5.1/kong/plugins/prometheus/api.lua:6>
15/03/2018 10:11:11stack traceback:
15/03/2018 10:11:11	[C]: in function 'error'
15/03/2018 10:11:11	/usr/local/share/lua/5.1/lapis/application.lua:397: in function 'handler'
15/03/2018 10:11:11	/usr/local/share/lua/5.1/lapis/application.lua:130: in function 'resolve'
15/03/2018 10:11:11	/usr/local/share/lua/5.1/lapis/application.lua:161: in function </usr/local/share/lua/5.1/lapis/application.lua:159>
15/03/2018 10:11:11	[C]: in function 'xpcall'
15/03/2018 10:11:11	/usr/local/share/lua/5.1/lapis/application.lua:159: in function 'dispatch'
15/03/2018 10:11:11	/usr/local/share/lua/5.1/lapis/nginx.lua:215: in function 'serve_admin_api'
15/03/2018 10:11:11	content_by_lua(nginx-kong.conf:150):2: in function <content_by_lua(nginx-kong.conf:150):1>, client: 10.42.150.236, server: kong_admin, request: "GET /prometheus/metrics HTTP/1.1", host: "api-admin.dev.mydomain.com"

Questions on functionality

So this plugin exposes metrics for Prometheus to pull correct? Does it expose the metrics on top of each api at a specific mykongproxy.com/my/proxy/endpoint/prometheus/metrics or something of that sort? As a WIP what is left to work out? My team is interested in Prometheus for Kong and may end up working on something similar.

"WIP - not ready yet"

Possible to create a milestone to track what is needed to make it ready for a release?

I can help out as well as this is pretty important for us.

Publish to luarocks

When releasing the project we will need to publish it to luarocks in order to make it available easyly for everyone.

http_upstream_duration_ms_histogram_sum value is -1

After taking plugin version v0.1.1 into use I have seen below errors from Kong container log.

06/06/2018 07:55:192018/06/06 07:55:19 [error] 51#0: *2715 [lua] prometheus.lua:296: log_error(): Error while setting 'http_upstream_duration_ms_histogram_sum{api="XXXX"}' to '-1': 'Value should not be negative', context: ngx.timer, client: 10.130.100.241, server: 0.0.0.0:8000
06/06/2018 07:55:1910.130.100.241 - - [06/Jun/2018:07:55:19 +1000] "GET /health HTTP/1.1" 200 5 "-" "ELB-HealthChecker/2.0"
06/06/2018 07:55:192018/06/06 07:55:19 [error] 51#0: *2717 [lua] prometheus.lua:296: log_error(): Error while setting 'http_upstream_duration_ms_histogram_sum{api="XXXX"}' to '-1': 'Value should not be negative', context: ngx.timer, client: 10.130.100.241, server: 0.0.0.0:8000
06/06/2018 07:55:1910

Seems it only happens on one specific api, and kong_http_request_duration_ms_gauge for that api is always 0

I have not checked deeply why it happened, just wondering first if you have seen this as well.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.