Some timestamped records might reach the Data Historian with high latency (lag of several minutes / days / years). These records should integrate seamlessly in the indexation workflow, they they should be eventually added to the relevant chunk.

For a given timestamps range, should we consider only one chunk, or is it better to handle multiple chunks ?

We need to package nice looking, and if possible branded, grafana dashboards in the distribution. This dashboards will also help in the documentation

It should be possible to add points through the REST API.. something like...
https://:XXX/historian-server /v1/points/create
{
"TagName": "openSpaceSensors.Temperature",
"points": [
{

            "TimeStamp": "2020-02-17T20:03:10.000Z",
               
            "Value":   "25",
               
            "Quality": 3
            }
       ]  

}

When re-compacting existing data with even possibly "late records", there is no impact on queries - that is to say, a new index or part of it might be - like in open analytics - created and when compaction is done and data clean, the newly created index replaces the online one.

To ease simple developments we need simple tar.gz we can just unzip. Docker is fine but not all customers want docker images for security reasons, more other docker requires some knowledge. A simple pre-configured tar.gz is fine with simple start/stop services.

A mechanism to export data should be available. Preferably through a REST API.

For audit and monitoring purposes we need a journal of activities. Especially we need to be able to see if compaction jobs are stucked for any reason (due to unhealthy Solr for example)

Please provide more detail... At least the type of api Request that cause this to reproduce the problem

to be discussed

Kura offers most of the connectors we need to integrate the factory to our data historian. We can document the use of Kura and extend Kura if needed with some components we have if needed. This issue depends on the Issue for adding data points through the historian server REST API.

A scalable compaction daemon should compact uploaded data on the fly

Current behaviour

the search endpoint currently does a contain query depending on request input.

Expected behaviour

The seach endpoint response should be more sophisticated so that it returns depend on input in a more flexible way, for exemple for the request :

{ target: 'upper_50' }

the answer should return only metric that seem's like upper_50.

["upper_25","upper_50","upper_75","upper_90","upper_95"]

and not only name containig the input

["upper_501","aupper_50","upper_50"]

Be able to define custom workflows to process data stored in the Historian, and store back this processed data in Historian.
(cf "Tasks" in InfluxDB : https://v2.docs.influxdata.com/v2.0/process-data/ )

adapt the loader to be ablt to load the IT use case data

A re-engineering of the engine should allow to rely on a direct connection to the historian server to push time series and allow logisland real time processors to be pluggin the historian server directly. This means changing the class loading mechanism in LogIsland. A a matter of fact this issue has to be pushed onto LogIsland but is there for tracking purposes

analyse SAX string and check if there's a 2 or 3 letter drop and add a tag accordingly with correct timestamp

The documentation needs simple CSV files for education purposes in using the historian. We need if possible data that we will use for testing purposes as well.

We need an intermediary storage for WAL informations (journal), if possible outside Solr, since Solr might be unavailable. A light java embedded database like Nitritre could be a possible way to go.

As of today, the real time part of the historian depends on the logisland engine which also depends on a message bus and in some cases with a strong dependency on Kafka. A re-engineering of the engine should allow to rely on a direct connection to the historian server to push time series and allow logisland real time processors to be pluggin the historian server directly.

Current behaviour

the search endpoint currently does not depend on user input. Moreover it returns all existing metrics.

Expected behaviour

The seach endpoint response should depend on input for exemple for the request :
{ target: 'upper_50' }
the answer should return only metric that seem's like upper_50.
["upper_25","upper_50","upper_75","upper_90","upper_95"]
and not
["hello","temp","pression"]

You can look at the documentation of the simple json plugin if you want more exemples :
https://grafana.com/grafana/plugins/grafana-simple-json-datasource

The max number of target returned should be configurable in the conf file of the Http verticle.

Which algorithm to filter results ?

• Only data starting by ?
• Only data containing the request ?
• The field name on which we make the request if of type String and contain a unique word.

Condition to close this issue

• implement a solution
• add an integration test

Document the deleteByQuery call on SolR to clean the data.

It should handle conflicting chunks. But the conflicting chunks should not be too degraded neither. Therefore we will suppose that this is not possible to have a chunk conflicting with 10000 others. If it was the case we would not guarantee anymore the query.

The API should encapsulate the Solr deleteByQuery to clean data in a clean and safe way.

As of today alerts are created via grafana and associated to graphs. We need to store alerts in the back end of the data historian especially since alerts are also created by the real time logisland part.

The Gateway is a bit misleading and this gateway is definitively our REST API to the back-end... so let's rename it to Historian Server

I would need to add tags that are not related to a particular timeserie but to the context of my Fab - for example an incident on a machine. This to be able to correlate problems. I would need to put an idenfier to relate my production machine and a bunch of sensors (in OPC this is a node in a tree).

Be able to define retention policies based on :

• measurement
• tags
• timestamps
• mixture of the above

We need to have a proper schema for all data historian concepts.

use that : https://github.com/prestosql/presto/tree/master/presto-parser

Maybe using a panel plugin ?

Currently annotation belong the grafana graphs and that works. But in the future we want annotations (as well as alerts) to be stored in proper solr collections

For exemple user should be able to chose for each metric :

• The sampling algorithm to use
• The size of buckets to use (but should we allow this parameter if it is conflicting with maxDatapoint of the graph ?)

I think if user specify a bucket size too small then the historian ignores it because it could freeze the historian in bucket size is not adapted to the number of points in the specified time range.

Annotation configuration for the plugin should be similar to the grafana builtin plugin. Being able to filter on tags.

On gateways side, it should support POST /annotations request with body like

{
  "range": {
    "from": "2016-04-15T13:44:39.070Z",
    "to": "2016-04-15T14:44:39.070Z"
  },
  "rangeRaw": {
    "from": "now-1h",
    "to": "now"
  },
  "limit" : 100,
  "tags": ["tag1", "tag2"]
  "matchAny": true,
   "type": "tags"
}

And should respond with something like

[
  {
    "time": 1581075145188,
    "timeEnd": 1581075145188,
    "text": "bbbb",
    "tags": [
      "tag1"
    ]
  }
]

Here are the specifications :

• "time" is required
• "timeEnd" is optional (this is used only for events in a range)
• "text" is required, it describe the event
• "tags" can be empty, but all tags should be returned if there is any

So those four fields should be save in solr documents for now in a separate collection "annotations".
The historian would return them in the expected response format when it receive the corresponding request.

Here a description of the fields of the request :

• "range" and "rangeRow" describe the time range in which we want to find annotations. Please use the same method to extract time as done for other endpoints like /query for consistency of the code. SO we should only return annotations whose "time" is in the requested range.
• "limit" to limit the max number of annotations to return.
• "tags" if the request "type" is "tags" this is used to filter annotation by tags otherwise it is not used.
• "matchAny" : if true, we should return any annotation containing at leas one of the tags. If false we should return only annotation containing all the tags.
• "type" : It is either "tags" either "all". tags type means we want to filter by tags, "all" type means we will return all annotations.

The solr schéma should be :

• time : long
• time_end : long
• tags : string multivalued
• description : text

Condition for closing this issue :

• Implement the /annotations endpoint according to the specifications
• Add integrations tests
• Test the solution with grafana

We need to design non specific spark jobs or java jobs for most common tasks that can run in standalone as well as distributed mode and can face massive amounts of data as well as small amounts of data.

We should be able to have the data historian made available on a single node with a Solr, the historian server, the Solr back-end and grafana for visualisation. All other components like Kafka, Spark, LogIsland should only be installed for very large volumes and real time or very advanced analytics

We should be able to create/update/delete alerts through the data historian server API

The grafana plugins have to be available in the tar.gz as well (separate or not) and their deployment should be documented.

Generally sensors data come with the level of quality. This quality should be stored along the values and possibly compacted in the same way

Follow this tutorial : https://medium.com/@ivanahuckova/how-to-contribute-to-grafana-as-junior-dev-c01fe3064502
I recommend using visual studio code and install the go plugin :

• https://code.visualstudio.com/
• https://code.visualstudio.com/docs/languages/go

Once the setup is done we will be able to make our own Datasource plugin based on the simple json datasource plugin.

For this we forked the grafana repo (under Hurence user). So if you have already made the tutorial with grafana repo you can follow this tutorial to change remote url to hurence's one : https://help.github.com/en/github/using-git/changing-a-remotes-url

Currently the only way to have data injected to the historian is through either big batch spark processes or with real-time injection through Kafka/LogIsland. A simple mechanism to import CSV files / Excel files should be available. Also may be a REST API to interact with (the current gateway is only for consuming data)

After this first release we need to have our data format versioned to ensure backward compatibility in upgrades

InfluxDB handles duplicates as a union of the two fields, keeping the latest value(s) indexed as well as previous fields values when they are not present in the last record

cf : https://v2.docs.influxdata.com/v2.0/write-data/best-practices/duplicate-points/

Depending on our record design, we should find a way to handle the duplicates properly

Real-time indexation of timestamped records stores unitary records in Solr (one value per record). This is useful to give access to these records in real time.
But there is an underlying need to compact these records into chunks on a regular basis for performance and storage concerns.

The compaction job steps :

• read all the relevant unitary data from Solr
• process this data into chuncks
• delete the unitary data and inject the corresponding compacted data. This operation must be Atomic to prevent from any discrepancy in the indexed data

This job should eventually be compatible with other time series backends (other than Solr)

The trigger to run the compaction job should be configurable, based on :

• time (every X minutes)
• number of records (every X records)
• number of records per partition (every X records per partition)
• a mixture of previous criteria

A Record should be composed of :

• "measurement" : a label (ex : "temperature" or "sensor_name", ...)
• 0**n tags : "0 to n" couples (tag_name, tag_value) used to index data
• 1**n fields : "1 to n" couples (field_name, field_value) to store actual measured values

(cf Line Protocol of InfluxDB : https://v2.docs.influxdata.com/v2.0/reference/syntax/line-protocol/ )

Current behaviour

The metrics name returned contain duplicates.

Expected behaviour

When I query on /search the response should not contain duplicate name !

How to reproduce

Insert several chunks with the same name. Query search andpoint the response will contain duplicates.

Condition

Fix and test that

• chunk_tags field (array to be specified with gateway team)

Since we want annotations to be stored in the back end, it should be possible to create/update/delete annotations using the REST API of the data historian

The rest API can take CSV files and a daemon will shunk/compact them prior to storage. This is a simple way to feed the data historian, pretty universal and quick for a first level of integration.

Handle multiple-values records : be able to store several values in a given record (one timestamp). Examples :

• raw value of the sensor + the quality of the measure
• https://docs.influxdata.com/influxdb/v1.7/concepts/key_concepts/#sample-data

( cf issue #13 as well )