joergen7 / cuneiform-legacy Goto Github PK

It should be possible to create foreign code in any given scripting language. Consider the example below:

declare python-example;

deftask + in python( ~result : ~<operand> ) *{
    #!/usr/bin/env python

    result = 0
    for line in operand:
        result += float( line )
}*

target x;

x = apply(
    task: +
    operand: ~'5' ~'10' ~'-7' ~'1'
);

• Bootstrapping
• Expectation Maximization
• Simulated Annealing

Currently, when an error occurs in a Bash script, it may be, that the workflow just continues.

It collects the error messages, but does not halt.

Make the log a bag of JSON objects. In the following, it might be useful to create a KNIME node that extracts a table out of the log.

Caching obsoletes the task ids and invocation ids since these can be different for multiple workflow run. Log instead the cacheIds.

This also makes it easier to find the results in the file system.

To ease the parse of a possible provenance software, input and output declarations should have the same form.

fa = apply(
task : connect
...
ext : fa
);

Invocation signatures must reliably represent invocations.

• if a task body is altered, the signature is altered too
• if an invocation signature changes, all downstream signatures also change
• use long signatures
• before any intermediate results are reused, compare all logs with all created directories and delete any directories, that aren't mentioned in any log

In order to allow provenance-explicit filters we need to be able to produce the empty list as a task result.

However, as task input it shouldn't be allowed.

What macro hasn't been found?

Is importing macros erroneous.

Also in the visualization, only relevant nodes should be drawn.

Right now, the LocalDispatcher filters the report entries for the invocation output. This can be pushed into the Invocation class itself.

Each task invocation writes things to the stdout and the stderr. Instead of just losing this information, the controlled side-effects of the invocation should appear in the controlled side effect of the workflow execution: the worklfow trace.

Task nodes have ids which are stored in the log. Since these ids may not be the same in a subsequent run, all task node ids have to be exchanged for signatures, as done for invocations.

Check whether all occurrences of task node ids can be replaced by signatures.

Packages should have dependencies only in one direction.

• Find attipatterns in code
• Issue warning at compile time
• optimize task graph

Some people prefer include. Make them interchangeable. Also can an import eat a string expression instead of just an identifier.

Readlink, the tool that resolves the full file name of an output variable, does not check for the existence of a file. Funny errors result.

Conditionals should be first level citizens.

Types should be checked.

Literals must not anymore be decorated. It suffices to have the task definition prototypes typed.

A test case has been added. This is a crucial bug.

Assigning the same variable multiple times should be forbidden, because all data types are immutable by definition.

By default, store the log in a separate file in the build directory.

Task names can be displayed inside the boxes if there is only one of them.

Data items don't have a type, variables in a prototype do have. Don't confuse them.

One gets the following error message when starting a task body with a tab:

missing BODY at '*'

Macro calls should look more like apply calls. Thus

• keyword arguments
• allow expressions, not only ids

• Altering the body of a function must also alter its hash.
• Altered intermediate data should affect downstream invocations.
• If an invocation did not produce an output it has to be recomputed.

This is especially important when workflow leaves are heavily altered. Because the current state forces the user to delete all cached results prior to his next run.

i -> __I

The invocations that are supposed to be started after an output reduce task, are never invoked.

Cuneiform has support of higher order functions. Consider the following example

declare higher-order-fn;

deftask calculate-fn( fn : ) {
echo my-task > $fn
}

deftask my-task( out : a b ) {
cat $a $b > $out
}

target x;

x = apply(
task: apply( task: calculate-fn )
a: ~'hello'
b: ~'world'
);

Use declare statement to create .dot graphs with meaningful names.