Comments (1)
Related to #22 and #10
A way of implementing compaction would be to use two différent storage.
- A heavy storage which could store historical data (called S0)
- A light storage that would store more recent data (called S1)
In this configuration we would guarantee that :
- all data in S0 are compacted efficiently without any intersection between chunks of a specific metric.
- whereas data in S1 are chunk of undefined size (may be one point), furthermore several chunk may be intersecting each others.
- The only process that writes data in S0 is the compaction job. All others process writes to S1.
A compaction job should then consist of those steps :
- Get all data in S1.
- Get all data in S0 which are intersecting with S1 data.
- Recalculate chunks with an algorithm to define.
- Replace all old chunks with new calculated ones.
The historian server would return data from request using both S0 and S1.
In this case the response would be guranteed to be accurate except during between step 4, during step 4 there may be some duplicate points in new compressed chunks and not yet deleted old unoptimized chunk in S1.
That's why in this case I would recommend to offer at least two modes for requests to users.
- A mode where we only query data on S0 to be sure of the accuracy of the data, but in this case we may not have more recent updates. (called for example ACCURACY)
- A real time mode where we may return duplicates but we got recent updates. (called REAL_TIME)
I think for a first implementation, this a good compromise between accuracy and dev effort.
But this design would impact features that update or create data.
For exemple issues #37 would probably writes into S1.
While issue #29 may writes directly on S0 ? Anyway this should be thought carefully when implementing feature that modify timeseries.
from historian.
Related Issues (20)
- improve chunk compaction
- make scaphandre integration
- make yarn intégration
- make spark history server integration
- add rate limiting to endpoints queries
- don't store quality aggregates in chunks if not needed
- modify historian-server startup script
- add greensights-server launcher
- make POST /api/v1/web-analytics/compute reactive
- improve CPU utilisation in greensights server
- add web analytics computation cron conf HOT 1
- add user/org management to historian
- add an auto-discover infrastrucutre tool
- add new KPI to greensights browser extension
- improve greensights server performance
- add an endpoint for getting old reports and page analytics
- detect page structure changes
- Create <aggregation>_over_time() functions
- Create topk operator
- docker image 1.3.10 not working
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from historian.