So that out_type is ordered and from there all generated code doesn't have to be aware.

This single text would define several nodes, optionally named.

Require that previous is no longer an alias for group.previous.

So that we could replace the whole top table in the GUI with a normal chart, alert on stale data, amongst other things.

To be used with the docker image.
Report that the daemon is running (obviously) and may include number of workers etc.

with configurable cache size.

If all parents have the same event definition, and all the fields used in this definition are also present in the child, and no specific event description is given, then propagate it.

We do not want to propagate the export flag though, so this needs to be separated from the event time description.

In.next and group.next tuples would come very handy for some non trivial commit clauses.

If In.next is used then we would wait for that tuple before processing the input tuple.
If group.next is used then we would just store the in tuple in the aggregate without further processing, and wait for the next one to take action. Instead we would process the last stored tuple as if it was the new one.

Depends on #360

Instead, either: broadcast the alert to all teams or force an editable flag "default" to one team?
Broadcasting seems to summon all kind of problems as it corner case the data model.

BCN will require #33 to compute full traffic volumes.

Many operations differs just by some constant parameters. If we generated code that obtained those values from the environment rather than hardcoding them, then we could reduce drastically the number of distinct binaries in many cases.

Possible approaches:

1. Mark such parameters in the operation text, for instance: WHERE foo=$p FOR PARAMETER p=42

2. Define template operations (or functions) in a specific, distinct phase: DEFINE FOO_FILTER(p) AS WHERE foo=$p and then set the operation text as FOO_FILTER(42)

3. More radically, make all immediate value a parameter.

This later approach is appealing since it does not involve the user at all and might reuse binaries that would not have been noticed as reusable otherwise. But on the other hand it can affect performance as it is no longer possible to optimise for known constants.

1. separate export flag from event time info
2. have export flag be changeable without recompilation (easy since we just have to edit the out_ref file)
3. ramen should then turn it on/off when timeseries are requested, and timeout after a while
4. a keyword to explicitly ask for a node result to be saved?

It should not be the case ATM that we modify them simultaneously but better safe than sorry.
It's a good way to check this assertion BTW.

Also, we will need this later for a fork-based implementation, anyway.

Repeat until received acknowledgment/0 exit status.
Repeated notifs must be stored elsewhere than in the ringbuf and persisted on disk.

Also solves #127 :

0. Each notification must have an identifier and a firing bool. We could add these to the generic notification and consider the others as fire-and-forget; So the identifier would be the name and we must add the firing boolean to the notify_cmd.
1. When a firing notification is received, it is timestamped (both with receive time and a schedule time) and saved in a heap (ordered by schedule time), which is immediately serialised on disc;
2. When a non-firing notification is received, look in the heap for this notification (requires to have an identifier for any notif: the keyword should provide it (as a string), as well as the firing bool for this to work) and cancel it. If not found, just ignore;
3. While waiting for new notifications, check the heap top notifs for a notification to schedule (starting from top, look for the first one not sending);
4. When eventually sending the heap top, leave the notification on the heap but flag it as sending. On success remove it and save the heap. On failure reschedule it and save the heap;
5. At start up, read the heap from disk and reset all sending flags.

For percentiles, predictors (predicted value + intervals), etc...

The recent WHERE patch have shown that we needed a distinct global state for the WHERE clause.
Maybe we could generalize and have one global state for all input, one for all selected, one for all unselected, and one per group? The clause would tell which default to use.

When missing some parameters the HTTP GET should return a form.

Do not stop nodes that are not changing though.

So that we can plot them as annotations over the graphs.

Whenever possible (ie we cannot enlarge a node from another layer).

Given a column that's supposed to be synchronized in N input streams, merge sort them into a single output (using the select clause to construct the output tuple).
For each next value of the synchronized column, build out using the last in of each input stream, then replace the input tuple of one of the input to reach the next smallest synchronized value (or advance several input streams if the same sync value is present in several of them).

This should be an extension to the normal Group-By operation as opposed to a new kind of operation.

Fixed configuration mode for the alerter where we specify the config.json to start with on the command line.

To not start/stop an incident if the alert is flapping.
Duration of these hystereses TBD.

A single listener on port 2003 must be able to fill several tables.
We could restrict to one table per metric (with additional fields from the tags, as factors).
Function name would be the metric path but the last few components, and main field name would be that last path components. How many components to chop off the metric path is unclear though, and should be part of the schema.
As tags are not necessarily present on every messages the schema will have to be declared by the workers wishing to convert some graphite metrics into a tuple stream, anyway.

So:

• a new command ramen graphite that starts a TCP forking server and a UDP listener on the given ports;
• collected graphite metrics should be enqueued without further ado into a #graphite ringbuf;
• then it should be possible to LISTEN FOR GRAPHITE $metric_prefix (...schema...) that would perform a pivot.

Alternatively, we do the same as what we do for collectd, using a simple tuple type of reception time, sender, metric name, tags (as a list), value and event time.

Would also help declaring start/stop/durations, as we could annotate a field with what type of time it represents (event start / event stop / event duration).

Not sure if we don't want instead some semantic attached to the type, that could be used to parse/print also data volume units, for instance, without requiring a whole new type?
Would probably depend on how light these new types can be made and how convenient they are to use as types.

Timeseries selects the tuple range using time only.
For this we need, in addition to the time range to select a single batch of TOP output.
So we need either a way to recognize such a batch or to provide a key that would be used to retrieve only the tuples from this time range with a single value for that key (so that we could reuse it for other things than tops).

Then a dedicated visualizer.

Once we are confident the skiplist works properly.

and visualizer.

With this, and once #21 is in place, then we could render incident chronology over any graph.

It might be more useful to sort the input rather than the output. So have one optional sorting clause per input stream?

Idea: have only one over N shard in RAM (both aggregates and the tuple queue) and the other N-1 on disk, along with N tuple queues.
When a tuple for another shard is received it's batch-queued on disc.
When we rotate the shards we load the queue and process it while newly received tuple for that shard are enqueued. So that we still process them in order.
On disc queues could be growable ringbuffers, as the one that we could also use for node history.

Requires to get rid of serialization and work on mmapped files for the groups.

Since we fail out of the compilation we do not save the resulting graph, therefore the last error is not recorded in the node.

Alternative design (that would still require a rw-lock to protect the file) would be for conf to be a persistent data structure. Compile would return a new one with either new graph or new error messages.But then we would have to catch exceptions when calling compile and return a conf indicating of the error rather than an http error code.

would be particularly pretty with a map visualizer.

To de-uglify the code and avoid mixing lwt with exceptions.

Need to use actual threads instead, assuming we do not need to parallelise anything that's blocking.

• compile with threads
• update RamenRWLock
• update RamenOutRef
• update RingBufLib
• everything else should follow rather easily
• regarding cohttp server, either wrap it, or replace it with a custom http server (à la csview), or replace it with a CGI interface and a lightweight external http server
• regarding cohttp client in notifier, either wrap it or replace it with ocamlnet or exec shell.

async threads to be replaced:

• watchdogs -> another thread, or a small filesystems monitored by the supervisor (touching a file per worker, where suspending the watchdog is replaced by deleting that file);
• workers reporting stats every X seconds -> easier from a dedicated thread;
• workers merging several ringbuffers with foreground wait times -> still need a background thread;
• wait_all_pids thread should not be needed any longer -> easier without threads;
• ramen tail reseting export timeouts on a timely basis -> no real need for a thread;
• notifier scheduler -> still better with a separate thread;
• notifier asynchronously sending notifications -> anything IO is easier with LWTs;
• tester threads in RamenTest to be replaced by actual posix threads.

If we start with the workers, we need to focus on watchdogs, reports, and merge.
Watchdogs are problematic as they are used by workers and ramen cli tools such as supervisor and notifier. We could have a mutex and a LWT version of watchdogs?

Will also serve as a confirmation dialog.

In a CASE or in OR/AND operations, if the operand is a test for nullability then we should propagate the knowledge that something is null or not down the AST. We could then better type operations on this operand; especially if it's known to be not null.

For instance:

if (a is not null && b is not null) then remember a || remember b

currently the state of the 2 remember operations will be an optional bloom filter, and the code will match against Some a and Some b; but all this is useless since we know that a and b will, here, always be non null.

What is needed is a special operation from nullable to not nullable (akin of Option.get) that would be automatically added around all usage of a and b in the branches, before the actual typing start.
Given this operator, this is a pure rewrite operation.

As long as the parent nodes do output the fields that this node is using.

We could cram all output tuples in the child input ringbuf, provided:

1. There is be a prefix to each serialized tuple to say where it is coming from;
2. There are as many read_tuple functions as there are parents;
3. All those read_tuple functions output the same in_tuple made of only the fields that are
   mentioned (in the order they are mentioned).

Alternatively, we could have several input ringbufs so no prefix required.

Alternatively, the parent could strip down its output to match a child input. This might not be as stupid as it sounds, since we already update its out_ref ; we could add a format spec in that file (a bit mask of the fields) and then, provided fields are ordered by name in encoded tuples, then the single write out tuple function could just skip unwanted tuples. This would come handy for exporting data, as we could enable it on a per field basis.