hurence / historian Goto Github PK
View Code? Open in Web Editor NEWbig data timeseries historian
License: Apache License 2.0
big data timeseries historian
License: Apache License 2.0
Some timestamped records might reach the Data Historian with high latency (lag of several minutes / days / years). These records should integrate seamlessly in the indexation workflow, they they should be eventually added to the relevant chunk.
For a given timestamps range, should we consider only one chunk, or is it better to handle multiple chunks ?
We need to package nice looking, and if possible branded, grafana dashboards in the distribution. This dashboards will also help in the documentation
It should be possible to add points through the REST API.. something like...
https://:XXX/historian-server /v1/points/create
{
"TagName": "openSpaceSensors.Temperature",
"points": [
{
"TimeStamp": "2020-02-17T20:03:10.000Z",
"Value": "25",
"Quality": 3
}
]
}
When re-compacting existing data with even possibly "late records", there is no impact on queries - that is to say, a new index or part of it might be - like in open analytics - created and when compaction is done and data clean, the newly created index replaces the online one.
To ease simple developments we need simple tar.gz we can just unzip. Docker is fine but not all customers want docker images for security reasons, more other docker requires some knowledge. A simple pre-configured tar.gz is fine with simple start/stop services.
A mechanism to export data should be available. Preferably through a REST API.
For audit and monitoring purposes we need a journal of activities. Especially we need to be able to see if compaction jobs are stucked for any reason (due to unhealthy Solr for example)
Please provide more detail... At least the type of api Request that cause this to reproduce the problem
to be discussed
Kura offers most of the connectors we need to integrate the factory to our data historian. We can document the use of Kura and extend Kura if needed with some components we have if needed. This issue depends on the Issue for adding data points through the historian server REST API.
A scalable compaction daemon should compact uploaded data on the fly
the search endpoint currently does a contain query depending on request input.
The seach endpoint response should be more sophisticated so that it returns depend on input in a more flexible way, for exemple for the request :
{ target: 'upper_50' }
the answer should return only metric that seem's like upper_50.
["upper_25","upper_50","upper_75","upper_90","upper_95"]
and not only name containig the input
["upper_501","aupper_50","upper_50"]
Be able to define custom workflows to process data stored in the Historian, and store back this processed data in Historian.
(cf "Tasks" in InfluxDB : https://v2.docs.influxdata.com/v2.0/process-data/ )
adapt the loader to be ablt to load the IT use case data
A re-engineering of the engine should allow to rely on a direct connection to the historian server to push time series and allow logisland real time processors to be pluggin the historian server directly. This means changing the class loading mechanism in LogIsland. A a matter of fact this issue has to be pushed onto LogIsland but is there for tracking purposes
analyse SAX string and check if there's a 2 or 3 letter drop and add a tag accordingly with correct timestamp
The documentation needs simple CSV files for education purposes in using the historian. We need if possible data that we will use for testing purposes as well.
We need an intermediary storage for WAL informations (journal), if possible outside Solr, since Solr might be unavailable. A light java embedded database like Nitritre could be a possible way to go.
As of today, the real time part of the historian depends on the logisland engine which also depends on a message bus and in some cases with a strong dependency on Kafka. A re-engineering of the engine should allow to rely on a direct connection to the historian server to push time series and allow logisland real time processors to be pluggin the historian server directly.
the search endpoint currently does not depend on user input. Moreover it returns all existing metrics.
The seach endpoint response should depend on input for exemple for the request :
{ target: 'upper_50' }
the answer should return only metric that seem's like upper_50.
["upper_25","upper_50","upper_75","upper_90","upper_95"]
and not
["hello","temp","pression"]
You can look at the documentation of the simple json plugin if you want more exemples :
https://grafana.com/grafana/plugins/grafana-simple-json-datasource
The max number of target returned should be configurable in the conf file of the Http verticle.
Which algorithm to filter results ?
Document the deleteByQuery call on SolR to clean the data.
It should handle conflicting chunks. But the conflicting chunks should not be too degraded neither. Therefore we will suppose that this is not possible to have a chunk conflicting with 10000 others. If it was the case we would not guarantee anymore the query.
The API should encapsulate the Solr deleteByQuery to clean data in a clean and safe way.
As of today alerts are created via grafana and associated to graphs. We need to store alerts in the back end of the data historian especially since alerts are also created by the real time logisland part.
The Gateway is a bit misleading and this gateway is definitively our REST API to the back-end... so let's rename it to Historian Server
I would need to add tags that are not related to a particular timeserie but to the context of my Fab - for example an incident on a machine. This to be able to correlate problems. I would need to put an idenfier to relate my production machine and a bunch of sensors (in OPC this is a node in a tree).
Be able to define retention policies based on :
We need to have a proper schema for all data historian concepts.
Maybe using a panel plugin ?
Currently annotation belong the grafana graphs and that works. But in the future we want annotations (as well as alerts) to be stored in proper solr collections
For exemple user should be able to chose for each metric :
I think if user specify a bucket size too small then the historian ignores it because it could freeze the historian in bucket size is not adapted to the number of points in the specified time range.
Annotation configuration for the plugin should be similar to the grafana builtin plugin. Being able to filter on tags.
On gateways side, it should support POST /annotations request with body like
{
"range": {
"from": "2016-04-15T13:44:39.070Z",
"to": "2016-04-15T14:44:39.070Z"
},
"rangeRaw": {
"from": "now-1h",
"to": "now"
},
"limit" : 100,
"tags": ["tag1", "tag2"]
"matchAny": true,
"type": "tags"
}
And should respond with something like
[
{
"time": 1581075145188,
"timeEnd": 1581075145188,
"text": "bbbb",
"tags": [
"tag1"
]
}
]
Here are the specifications :
So those four fields should be save in solr documents for now in a separate collection "annotations".
The historian would return them in the expected response format when it receive the corresponding request.
Here a description of the fields of the request :
The solr schéma should be :
Condition for closing this issue :
We need to design non specific spark jobs or java jobs for most common tasks that can run in standalone as well as distributed mode and can face massive amounts of data as well as small amounts of data.
We should be able to have the data historian made available on a single node with a Solr, the historian server, the Solr back-end and grafana for visualisation. All other components like Kafka, Spark, LogIsland should only be installed for very large volumes and real time or very advanced analytics
We should be able to create/update/delete alerts through the data historian server API
The grafana plugins have to be available in the tar.gz as well (separate or not) and their deployment should be documented.
Generally sensors data come with the level of quality. This quality should be stored along the values and possibly compacted in the same way
Follow this tutorial : https://medium.com/@ivanahuckova/how-to-contribute-to-grafana-as-junior-dev-c01fe3064502
I recommend using visual studio code and install the go plugin :
Once the setup is done we will be able to make our own Datasource plugin based on the simple json datasource plugin.
For this we forked the grafana repo (under Hurence user). So if you have already made the tutorial with grafana repo you can follow this tutorial to change remote url to hurence's one : https://help.github.com/en/github/using-git/changing-a-remotes-url
Currently the only way to have data injected to the historian is through either big batch spark processes or with real-time injection through Kafka/LogIsland. A simple mechanism to import CSV files / Excel files should be available. Also may be a REST API to interact with (the current gateway is only for consuming data)
After this first release we need to have our data format versioned to ensure backward compatibility in upgrades
InfluxDB handles duplicates as a union of the two fields, keeping the latest value(s) indexed as well as previous fields values when they are not present in the last record
cf : https://v2.docs.influxdata.com/v2.0/write-data/best-practices/duplicate-points/
Depending on our record design, we should find a way to handle the duplicates properly
Real-time indexation of timestamped records stores unitary records in Solr (one value per record). This is useful to give access to these records in real time.
But there is an underlying need to compact these records into chunks on a regular basis for performance and storage concerns.
The compaction job steps :
This operation must be Atomic to prevent from any discrepancy in the indexed data
This job should eventually be compatible with other time series backends (other than Solr)
The trigger to run the compaction job should be configurable, based on :
A Record should be composed of :
(cf Line Protocol of InfluxDB : https://v2.docs.influxdata.com/v2.0/reference/syntax/line-protocol/ )
The metrics name returned contain duplicates.
When I query on /search the response should not contain duplicate name !
Insert several chunks with the same name. Query search andpoint the response will contain duplicates.
Fix and test that
Since we want annotations to be stored in the back end, it should be possible to create/update/delete annotations using the REST API of the data historian
The rest API can take CSV files and a daemon will shunk/compact them prior to storage. This is a simple way to feed the data historian, pretty universal and quick for a first level of integration.
Handle multiple-values records : be able to store several values in a given record (one timestamp). Examples :
( cf issue #13 as well )
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.