steep-wms / steep Goto Github PK

In daily use, there is often a sequence of services that is included in each of my workflows. For example, this can be copying data into my environment, or pre-processing before the actual calculation starts. However, at the moment I have to write these services every time into my workflow.

It would be nice if I could outsource these steps to a macro. Similar to the service.yaml there would then be a macro.yaml containing predefined sequences of services (i.e. more or less small workflows). They also have inputs and outputs so that I can continue working with the results in my actual workflow.

For example, a macro.yaml could look like this:

id: preparation
name: Preparation of data
inputs:
  - id: "source"
    var: "source"
outputs:
  - id: "destination"
    var: "destination"
actions:
  - type: execute
    service: download
    inputs:
      - id: "source"
        value: "source"
    outputs:
      - id: "output"
        var: downloadedData
  - type: execute
    service: preprocess
    inputs:
      - id: "input"
        value: "downloadedData"
    outputs:
      - id: "output"
        var: destination

This bundles the two services download and preprocess and makes them available as a macro preparation.
In my workflow I could then use it like this:

api: 4.6.0
actions:
  - type: macro
    macro: preparation
    inputs:
      - id: source
        value: http://example.com/my-data
    outputs:
      - id: destination
        var: preparedData
  - type: execute
    service: calculation
    inputs:
      - id: input
        var: preparedData
    outputs:
      - id: output
        var: outputDirectory

Can something like this be added to Steep?

Throw an error if the configuration contains duplicate setup IDs

It should be possible to override SSH settings from the global configuration per setup.

There should be an option that makes the Docker runtime call docker pull before docker run if the image tag is latest.

I can define plugins to adjust the execution behavior. It is possible to define multiple plugins of the same type. For example I can define two processChainAdapter plugins A and B.
Currently their execution order is given by the definition order in the corresponding yaml file. This is an implicit definition and might change at any time in the future.

It would be nice to have an order attribute for each plugins. Plugins with a larger order are executed after plugins with a lower order independently of their definition order in the yaml file.

Example:

- name: myPluginA
  type: processChainAdapter
  order: 2
  scriptFile: myPluginA.kt
- name: myPluginB
  type: processChainAdapter
  order: 1
  scriptFile: myPluginB.kt

In this case the plugin B should be executed before plugin A.