Giter Club home page Giter Club logo

ovh-warp10-datasource's Introduction

Grafana Warp 10™ Datasource Plugin

Install the plugin

Using grafana-cli

sudo grafana-cli plugins install ovh-warp10-datasource
installing ovh-warp10-datasource @ 2.2.0
from: https://grafana.com/api/plugins/ovh-warp10-datasource/versions/2.2.0/download
into: /var/lib/grafana/plugins

✔ Installed ovh-warp10-datasource successfully 

Restart grafana after installing plugins . <service grafana-server restart>

Cloning the repository

Just clone the repository in the Grafana plugins folder

git clone [email protected]:ovh/ovh-warp10-datasource.git /var/lib/grafana/plugins/ovh-warp10-datasource

Grafana will use the dist/ folder by default

Grafana 7.0 troubleshooting

As mentioned in #71, "CORS" issues can occurs with a Warp10 datasource. To fix it:

  • Or set x_content_type_options to false in Grafana servers configuration settings,
  • Or whitelist content-type in allow headers of Warp10 with the configuration key cors.header.

Add a new Warp 10™ Datasource

  • Go to the Grafana menu (top left) > "datasources" > "add data source"
  • Choose a name
  • Set Warp10 as type
  • Paste the Warp 10™ Platform exec endpoint URL ( do not append /api/v0/... )

Global Variables

Add execution variables

You can define variables at datasource level (~ organisation level) which can be available for all dashboards. you can put tokens, constants, macros, ... In case of a macro definition, the variable value must start with <% and end with %>. In the query you can prepend @ to the macro name to execute it.

For example, you can store a read token here:

Global Variables

Make a query

On a new dashboard, in a Graph visualization, click on Query icon on the left side bar, and choose Warp10 datasource.

A user friendly query editor will appear. You can use the global variable you defined previously:

First query

It will generate WarpScript for you. The generated WarpScript is visible in your browser console:

Console output

You will see at the beginning of the query some special variables set by the plugin.

Name Description Example
$end Timestamp of the most recent point in the Grafana time window 1498038153276000
$endISO end value in ISO-8601 format '2017-06-21T09:42:33.276Z'
$start Timestamp of the less recent point in the Grafana time window 1498034553276000
$startISO start value in ISO-8601 format '2017-06-21T08:42:33.276Z'
$interval Difference between $end and $start 3600000000
$__interval Could be used as a bucket span to display less datapoints on a graph, saving your browser RAM. See Grafana doc. 3600000000

Advanced users can check the WarpScript editor and write the WarpScript they need.

To graph something on Grafana you need to return some Geo Time Series™ (GTS)

Graph example

The plugin look for GTS or GTS array in your stack, all other stack entry will be ignored. Turn on the WarpScript™ editor, and use this script:

NEWGTS
'io.warp10.grafana.test' RENAME
{ 'func' 'sinus' } RELABEL
'sinus' STORE

NEWGTS
'io.warp10.grafana.testmetric' RENAME
{ 'func' 'cosinus' } RELABEL
'cosinus' STORE

100 // Not graphable -> ignored
'b' // Not graphable -> ignored

$interval 20 / TOLONG 'step' STORE

<% $step + %> 'stepMacro' STORE
<% 'index' STORE $sinus $index NaN NaN NaN $index SIN  ADDVALUE DROP %> 'execMacroSinus' STORE
<% 'index' STORE $cosinus $index NaN NaN NaN $index COS  ADDVALUE DROP %> 'execMacroCoinus' STORE

$start $end $stepMacro $execMacroSinus FORSTEP
$start $end $stepMacro $execMacroCoinus FORSTEP
$sinus $cosinus

sinus output

Table example

By default, the plugin build a table with the timestamp as the first column, and one column per GTS.

You can build custom tables instead of formating GTS array, if your result stack have only 1 element and this element have columns and rows property. Then you can choose Table as Table transform in Table Options section

WarpScript™ example:

{
  'columns' [
    {
      'text' 'columnA'
      'type' 'number'
      'sort' true
      'desc' true
    }
    {
      'text' 'columnB'
      'type' 'number'
    }
  ]
  'rows' [
    [ 10 20 ]
    [ 100 200 ]
  ]
}

table output

Worldmap example

You can use ovh-warp10-datasource as datasource for showing position data on grafana using grafana-worldmap-panel plugin.

In order to do it, you need to install the grafana-worldmap-panel plugin: Worldmap Panel.

When both ovh-warp10-datasource and the grafana-worldmap-panel installed, you can define a new Worldmap widget, with a Warp 10 datasource and json result as Location Data in the Worlmap tab:

Warp 10™ datasource

json result as Location Data in the Worlmap tab

Now in your WarpScript™ you can generate data in the JSON format supported by Worldmap, for example :

'[ { "key": "amsterdam", "latitude": 52.3702, "longitude": 4.8952, "name": "Amsterdam" }, { "key": "charleroi", "latitude": 50.4108, "longitude": 4.4446, "name": "Charleroi" }, { "key": "frankfurt", "latitude": 50.110924, "longitude": 8.682127, "name": "Frankfurt" }, { "key": "london", "latitude": 51.503399, "longitude": -0.119519, "name": "London" }, { "key": "paris", "latitude": 48.864716, "longitude": 2.349014, "name": "Paris" } ]'
JSON->

And then you can see the chosen locations in the map:

Worlmap view without values

You can also give a value to each location, in order to show the locations with different sizes and colors, as Worldmap allows:

'[ { "key": "amsterdam", "latitude": 52.3702, "longitude": 4.8952, "name": "Amsterdam", "value": 9 }, { "key": "charleroi", "latitude": 50.4108, "longitude": 4.4446, "name": "Charleroi", "value": 6 }, { "key": "frankfurt", "latitude": 50.110924, "longitude": 8.682127, "name": "Frankfurt", "value": 9 }, { "key": "london", "latitude": 51.503399, "longitude": -0.119519, "name": "London", "value": 12 }, { "key": "paris", "latitude": 48.864716, "longitude": 2.349014, "name": "Paris", "value": 15 } ]'
JSON->

Worlmap view with values

Use Annotations

You can add Annotation on your graph: Dashboard > "Manage dashboard" > "Annotations"

Just add you WarpScript™

/!\ You must return a single GTS on TOP of your stack

Annotation example

NEWGTS
'alerts' RENAME
{ 'a' 'b' 'c' 'd' } RELABEL
$end $interval 2 / - NaN DUP DUP 'Restart WebServer' ADDVALUE
$end $interval 3 / - NaN DUP DUP 'Update v1.0.2' ADDVALUE

annotation output

Define Templating variables

You can make a WarpScript query to build the choice list of your templating variables. In the dashboard settings, select Variables, and create a new one from a Query, with Warp10 as datasource. You can write any WarpScript in the Query field to build your list of choices:

annotation output

  • If you let several values on the stack, each value will be added to the choice list.
  • Best practice: Let a list on the stack. Each value will be added to the choice list.
  • Best practice: Let a map on the stack. The map keys will be added to the choice list, the map values will be available within the pannels WarpScript query. The values will be hidden from the dashboard user. This allow to hide complex values behind user friendly labels.

Here are valid queries for variable definition: annotation output

Each value is transformed into two WarpScript variables you can use in your queries:

  • A string, named as you named your variable.
  • A list of strings, named as you named your variable, suffixed by _list.

annotation output

  • If you do not use multiple selection, variable and variable_list will contain the currently selected value
  • If you use multiple selection:
    • the string will contain an optimized WarpScript regular expression
    • the list will contain each element selected
  • If you defined a custom all value and checked "All", variable and variable_list will contain your customized value.

In the example below,

  • testMap has a custom all value set to ~custom.*allvalue.*
  • testList has no defined custom all value.

annotation output

Query returning Labels

A variable can contain values for a defined Label. For example, to get all the unique values for the key hostname, you can specify a query like this in the templating variable Query setting.

[ $ReadToken '~.*' { 'hostname' '~.*' } ] FIND
<% DROP LABELS 'hostname' GET %> LMAP
UNIQUE

You can also create nested variables. For example if you had another variable, for example $region. Then you could have the hosts variable only show hosts from the current selected region with a query like this:

[ $ReadToken '~.*' { 'region' $region } ] FIND
<% DROP LABELS 'hostname' GET %> LMAP
UNIQUE

You can fetch keys for a given Class.

[ $ReadToken '<class_name>' { } ] FIND
<% DROP LABELS KEYLIST %> LMAP
FLATTEN
UNIQUE

Templating variable evaluation

To understand the variable resolution, this is how a query is built

  • Inject dashboard variables ($end, $interval, etc...)
  • Inject datasource execution variables (Customized by datasource)
  • Inject templating variables following the configuration order (a templating variable can call the previous templating variables in its resolution)
  • Inject user query (can use all previous variables)

/!\ all of the templating values are casted into strings by Grafana engine.

User friendly query builder

  • Basic Fetch
  • Bucketizer
  • Reducer
  • Renamer
  • Mapper
  • Filter
  • Extend limits (LIMIT, MAXOPS, MAXFETCH, ...)
  • Anomaly detection

Data fetching example

// you can use this public token on the https://warp.senx.io endpoint to fetch raw sensor data from a connected BeerTender.

"gCA1SVjbDkaxtmTx9ydI4TI2iGc5hFgcbCWnzMRZSt45XW8dZ53Z7VK_if28i0kXwNDLazXHgLrXUKgxLK0RbS79eJmBCpyBlIxw9US7bPfdWH4Fta51.kXN.D4Hsk5OZOwl.vLRBzMpP7F2pAMfclMXSGtCOT6F"
'rt' STORE


[ $rt '~beertender.(rawvalue|rawoffset)' {} $end $interval ] FETCH 
'rawResult' STORE

[ $rawResult [] 'beertender.rawvalue' filter.byclass ] FILTER 0 GET 'rawvalue' STORE

// do a linear interpolation to convert raw sensor value into percent.
// empty : 30.8e6
// full (cold): 30.52e6
30.8e6  $rawvalue - // you can substract GTS and constants
[ SWAP 0 mapper.max.x 0 0 0 ] MAP 0 GET // clamp min value to zero
2800 /  //divide by 2800
[ SWAP 100 mapper.min.x 0 0 0 ] MAP 0 GET // clamp max value to 100
[ SWAP bucketizer.mean 0 1 h 0 ] BUCKETIZE // keep one point per hour, the mean of each hour.


'level in percent' RENAME 

Related links

ovh-warp10-datasource's People

Contributors

aurrelhebert avatar bensmrs avatar d33d33 avatar dependabot[bot] avatar gmasse avatar hbs avatar leoncx avatar lostinbrittany avatar miton18 avatar nsteinmetz avatar pi-r-p avatar redref avatar stevenleroux avatar tomeros avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ovh-warp10-datasource's Issues

Grafana 7 removes error headers

image

This bug was found by LH on stack overflow, here.

I checked with the latest Warp 10 release, no problem with Grafana 6.
Then I checked with Grafana 7.0.0 and 7.3.6, something in the query processing is removing the headers from the error answers. So, the plugin is unable to display error and error line.

In the error handler, err do not contain anything usefull in Grafana 7, here in the code...

Error when adding query in Templating

When adding a query in templating we have this error :

Templating
Template variables could not be initialized: b.metricFindQuery is not a function

ES filters on tables breaks Warp10 annotations queries

When using a Warp10 datasource for annotations, creating filters in ES tables breaks the Warp10 query.

Warp10 annotation:

[ $token 'mySerie' { } $end $interval ] FETCH 

When creating a filter in the dashboard (here a table using an ES datasource):
create_filter

filter

The following JS error happens every time the dashboard is refreshed (and the annotation is not shown anymore):

AnnotationSrv.query error TypeError: "myVar.current is undefined"
    computeGrafanaContext https://GRAFANA_URL/public/plugins/grafana-warp10-datasource/warp10-datasource.js:254
    annotationQuery https://GRAFANA_URL/public/plugins/grafana-warp10-datasource/warp10-datasource.js:134
    s https://GRAFANA_URL/public/build/app.469095018b321ef1da7c.js:22
    c https://GRAFANA_URL/public/build/vendor.469095018b321ef1da7c.js:130
    c https://GRAFANA_URL/public/build/vendor.469095018b321ef1da7c.js:130
    $digest https://GRAFANA_URL/public/build/vendor.469095018b321ef1da7c.js:130
    $apply https://GRAFANA_URL/public/build/vendor.469095018b321ef1da7c.js:130
    l https://GRAFANA_URL/public/build/vendor.469095018b321ef1da7c.js:130
    h https://GRAFANA_URL/public/build/vendor.469095018b321ef1da7c.js:130
    n https://GRAFANA_URL/public/build/vendor.469095018b321ef1da7c.js:130
app.469095018b321ef1da7c.js:22:827595

There is no "exec" call made after in the browser Network inspector.

The error still happens after removing the filter from the dashboard with the following action:
remove_filter

and will only stop once the "Filters" variables (as shown bellow) is removed.

filters

Add prefix code to add to all requests

Hello,
It is possible to set variables in the warp10 data source that are available in all requests.
But it would be even better to be able to add any code (not just a variable) that would be inserted in all requests. For example I could add one line to raise the quota of number of points that can be fetched.

All variable not interpreted

When All is the default value, everything is fine.
When all is send a URL query parameter, it is not interpreted in Warp10 query

Annotation tags problem

On Grafana 5.4.3, the annotation query is expected to return the tags in a list, not in a comma-separated string. This change is documented in v6.1, but was implemented before.
I wrote a fix here:
bensmrs@93d23a3
but it shouldn't be backward compatible.
Would it be possible to check Grafana's version and format the tags accordingly?

Plugin not working with Grafana => 6.7

Hi,
The plugin needs some updating to work with Grafana version of 6.7 or higher as at the moment it does not work at all.
The error message is "err.headers is not a function"

Thanks.

Multiple and All values are send as 'value1 + value2 + …'

Hi,

When using multiple values or All options (with auto mode), the query sent to the Warp10 datasource is like that :
'value1 + value2 + value3 + …' variable STORE

Warp10 didnt like it. A best way will be to send something like :
'~(value1|value2|value3|…)' variable STORE

With All option, I can override with a custom value of '~.*' but it does not work with multiple option.

Dashboard templating

Hi,
When using warp10 query for templating purpose, the value of the var is the position of the element in the stack for the second element to the last one. I use grafana v4.
Regards,

dashboard.txt

Alias metrics by a label value

Use case : I want to display the traffic by campaignID ; so I use campaignID as the reducer and I get 10 series to display ; in the legend I want to name each metric with the campaignID value, but there is no way to do that with the "User friendly query builder".

To do this, I have to write it in Warpscipt and add :
<% DROP DUP LABELS 'campaignID' GET RENAME %> LMAP SORT

But Warpscript is less user friendly :-) and this is a very common use case ; in fact I do that for all graph panels.

Proposal : do it the same way than Prometheus with a field "Legend Format" where you can set exactly what you want to display ; {{campaignID}} means for example the value of the label campaignID

Angular is deprecated - please migrate to React

Hello 👋 from Grafana Labs

We have detected that your plugin might still be using AngularJS and recommend migrating it to React, so that it may continue to operate successfully in future versions of Grafana.

You can read more on why and how by visiting our documentation: https://grafana.com/docs/grafana/latest/developers/angular_deprecation/

You can reach out to us with any questions via slack: slack.grafana.com - channel #plugins or our plugin development community forum

Improper license

Following the transfer of ownership from Cityzen Data to OVH, the license has wrongly been changed from Apache 2.0 to a BSD like, this is in violation of the original license and is a change which must be reverted ASAP.

$__all trick breaks Repeat + Include All option

Hi Guys !

The recent addition of the "All option" handling, using the REOPTALT to generate a full matching regular expression, breaks the panels Repeating feature when the "Include All option" is selected without the "Multi-value" option:

Here is what we should get (Here, I'm using Repeating + Multi-value + Include All option):

Repeat + Multi-value + Include All option

And here is what's happening when I remove the Multi-value option (keeping just Repeating + Include All option):

Repeat + Include All option

The "Only queries that return single series/tables is supported" message comes from the REOPTALT generated regular expression which is present in every repeated panel. The consequence is I fetch the 4 hosts related GTS instead of the single one that is expected by the SIngle Stat panel.

Olivier

interval variable semantic

Hi,

according to grafana doc http://docs.grafana.org/reference/templating/#the-interval-variable the $__interval variable is a great as the bucket size parameter. That how it is used for influxdb and elasctisearch data sources.

but in warp, there is a variable interval injected into the warpscript but its equal to the current time range (if I select "last 3 hours' as a time range, interval=3h). So if I use it as the bucket size parameter, it yeld only 1 data point for the whole time range.

editor.js 404 Not Found behind nginx reverse proxy

Hello,

I Don't know if this issue concerns Grafana Team or Cityzendata.
I follow the doc : http://docs.grafana.org/installation/behind_proxy/
So, when I use warp10 plugin behind nginx revers proxy, the editor.js is not available because the url is not rewriting with '/grafana/' as configured in nginx before the domain name.

Unhandled promise rejection
Error: Fetch error: 404 Not Found
Instantiating http://myUrl.fr/public/plugins/grafana-warp10-datasource/editor.js
Loading /public/plugins/grafana-warp10-datasource/editor.js

good url is : http://myUrl.fr/grafana/public/plugins/grafana-warp10-datasource/editor.js

All others functionality of warp10 plug-in work fine.
Please, could you check if 'root_url' configuration variable is supported ?

thank you

Here my confs :
grafana.conf.txt
nginx.conf.txt

tried to use in Grafana 8.3

Hi, I tried to follow the examples in using world map with this plugin, however I encountered this error WarpScript Failure on Line -1, Unable to read x-warp10-error-line and x-warp10-error-line headers in server answer, does anyone encountered this ? I'm using grafana 8.3

Regards

Authentication token should be per datasource

The token used to authenticate against warp endpoint should be defined at the datasource level to avoid writing it in every warpscript. Because, if the token change, it would be very annoying to edit the tens/hundreds of scripts just to update the token.

The datasource should replace a well defined string in the warp script by the token.

Eg:

WARP script:
[ $warp_ds_token '=my.timeserie' { 'tag' 1 } $end $interval ] FETCH 'gts' STORE

The datasource would replace $warp_ds_token by the token.

It seems it could be easy to do in datasource.js, function metricFindQuery().

Warp10 datasource proxy access doesn't work with Firefox (Chromium is OK)

Hi Guys !

I have problems trying to use the proxy datasource access mode with Firefox.

Using Chromium (81.0.4042.0 (Build de développement) (64 bits)), I can access my datasource in both direct and proxy mode:

Chromium Direct
Chromium Proxy

Using Firefox (72.0.2 (64 bits)), direct mode is OK, but proxy mode shows a "res.err is undefined" error:

Firefox Direct
Firefox Proxy

Feel free to ask if you need any other information.

Olivier.

Panel & Row repeats

Given a dashboard with the following template variable:

  • name: foo
  • type: custom
  • values: 1,2
  • multi-value: true

When enabling panel repeat (or row repeat, same issue), the requests generated to fetch the data are invalid if there is multiple values selected (which is the point of the repeating option).

Expected:
If the two values of foo are selected, two requests must be generated.

  • first one should contain 1 'foo' STORE
  • second one should contain 2 'foo' STORE

Observed:
Two requests are generated, both with '1 + 2' 'foo' STORE

It works fine with OpenTSDB datasource, so I guess it's a datasource issue.

Make time range variables $end and $start available for in grafana templating variables

Use case: I want to create a templating variable 'active_issue' to allow the user to select an issue among the list of ACTIVE issues during the selected time range.

Problem: the time range selected - $start and $end variables that I can use in a panel - is unknown in the templating variable space. I can only select active issues NOW and not ofr the selected time range.

Request: make those variables available for templating variables.

Worldmap: Unsupported data format error

On a fresh install with last version of Grafana (6.7.3), worldmap panel (0.2.1)
I always get this error : Unsupported data format:
image

I tried with example data available on documentation and it also fails.

Plugin renaming

To be compliant with the official Grafana plugin registry, we will rename the plugin name.
cf. grafana/grafana-plugin-repository#302

grafana-warp10 => ovh-warp10-datasource

Be aware, if the name of the plugin change and you try to update an existing version of the plugin, you have to remove the old one.

Alerting Support

Will you add the support for the alerting function of Grafana in this datasource ?

How to use X/Y chart

I am not able to find the input for X/Y Chart.
I would like to use the X/Y chart to ploat 6 series of data.
Right now my output is the following one:
[
{ "label": "serie1Name" "values": [ [ 1 1 ] [ 2 2 ] ] }
{ "label": "serie2Name" "values": [ [ 10 10 ] [ 20 20 ] ] }
]

Thx for your help

Output examples for integration with graphs

Hello!

It could be interesting to provide examples to integrate with others graphs. For example, it may be something like this:

Here we will count the number of server per rack:

[ 'token' 'my.series' { 'rack' '~.*' } ] FIND

{} SWAP
<%
    LABELS 'rack' GET 'r' STORE

    <% $r CONTAINSKEY %>
        <%
            DUP $r GET 1 + $r PUT
        %>
        <%
            1 $r PUT
        %>
    IFTE
%> FOREACH 'm' STORE

{
  'columns' [
    {
      'text' 'Rack'
      'type' 'string'
    }
    {
      'text' 'server count'
      'type' 'number'
      'sort' true
      'desc' false
    }
  ]
  'rows'
    [ 
        $m KEYLIST
        $m VALUELIST
    ] ZIP
}

Update documentation

For the worldmap plugins, since the PR has been accepted and merged into the official grafana/worldmap-panel.

It would be great to rewrite this part of the README.md to have clearer instructions matching this changes.

Error while managing Warp10 datasource with Grafana 6.4.1

With a fresh install of Grafana (6.4.1), the warp10 datasource rises an error when accessing to the creation form:

Error: ace is not defined Evaluating https://.../public/plugins/ovh-warp10-datasource/warp10-query.controller.js Evaluating https://.../public/plugins/ovh-warp10-datasource/module.js Loading plugins/ovh-warp10-datasource/module
Data Source Error

image

Firefox N/A everywhere

With Firefox 75 (last version available), requests seems to always fail.
When I look in dev tools network information, I can see that Firefox does not see data as it should.
When I try to copy the request from dev tools as curl command and executes it from a terminal, it works fine.

With Firefox 75:
image

With Chrome:
image

I use charts and singlestats panels and I get this issue with both.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.