A compaction daemon should compact uploaded data on the fly about historian HOT 1 CLOSED

Related to #22 and #10
A way of implementing compaction would be to use two différent storage.

• A heavy storage which could store historical data (called S0)
• A light storage that would store more recent data (called S1)

In this configuration we would guarantee that :

• all data in S0 are compacted efficiently without any intersection between chunks of a specific metric.
• whereas data in S1 are chunk of undefined size (may be one point), furthermore several chunk may be intersecting each others.
• The only process that writes data in S0 is the compaction job. All others process writes to S1.

A compaction job should then consist of those steps :

1. Get all data in S1.
2. Get all data in S0 which are intersecting with S1 data.
3. Recalculate chunks with an algorithm to define.
4. Replace all old chunks with new calculated ones.

The historian server would return data from request using both S0 and S1.
In this case the response would be guranteed to be accurate except during between step 4, during step 4 there may be some duplicate points in new compressed chunks and not yet deleted old unoptimized chunk in S1.

That's why in this case I would recommend to offer at least two modes for requests to users.

• A mode where we only query data on S0 to be sure of the accuracy of the data, but in this case we may not have more recent updates. (called for example ACCURACY)
• A real time mode where we may return duplicates but we got recent updates. (called REAL_TIME)

I think for a first implementation, this a good compromise between accuracy and dev effort.

But this design would impact features that update or create data.
For exemple issues #37 would probably writes into S1.
While issue #29 may writes directly on S0 ? Anyway this should be thought carefully when implementing feature that modify timeseries.

from historian.