Draft 2 changes about common-workflow-language HOT 5 CLOSED

Not sure about stdout valueFrom - we already have the stdout property for that. Even the stdout flag - one can just set the glob to exact value or such.

We can't get rid of stdin/stdout properties cause their values might not be inputs/outputs.

Shell and shellquote seem quite useful. Can we extract this into a separate issue for spec discussion? Also seems it should be listed as extension/requirement.

The secondaryFiles might also deserve extension/requirement just to keep the core spec simple.

from common-workflow-language.

Not sure about stdout valueFrom - we already have the stdout property for that. Even the stdout flag - one can just set the glob to exact value or such.

There is a general use case for being able to construct the expected output file name based on the input. For example, one of the inputs to a tool specifies a "-o" (output) option and users may specify arbitrary file names for the tool output. In that case, it's impossible to reliably match the output correctly without referencing the input somehow.

I think the right solution is to involve expressions somehow (see next comment).

We can't get rid of stdin/stdout properties cause their values might not be inputs/outputs.

So in that case I would suggest we reject the proposal for a boolean on input/output bindings indicating stdin/stdout in favor of just keeping stdin/stdout properties on the tool itself; having two ways to specify the same thing just creates confusion.

Shell and shellquote seem quite useful. Can we extract this into a separate issue for spec discussion?

Agreed. Added #39 for shell/shellquote

The secondaryFiles might also deserve extension/requirement just to keep the core spec simple.

I disagree. File handling is a fundamental aspect of CWL since platforms are expected to manage what files are available to the tool. I actually intend to go further with the current reference implementation and bind mount individual files into Docker, so anything not explicit in the input or tool description is not available to the tool.

This will avoid situations where users make workflows that are subtly broken by assuming some secondary file X is next to the primary file and have it work on one platform that is sloppy about file handling (the tool gets the whole directory including files it didn't ask for) but fails on a different platform that is strict about file handling (that only provides the files explicitly requested).

from common-workflow-language.

On further thought, maybe outputBinding glob + loadContents + valueFrom (which is basically the current spec) is already sufficient to cover most of our cases? The outputBinding steps are applied in this order, and each step is optional (but at least one of glob or valueFrom must be provided):

1. glob selects files into a list of file objects, for example glob: *.txt yields [{"path": "foo.txt}, {"path":"bar.txt"}]
2. loadContents: true loads the first X bytes into the contents field for each file (optional) (maybe the value of loadContents should be the number of bytes instead of just a flag?)
3. valueFrom is an expression which takes the glob + loadContents as the context value and yields the final value.

If the output filename can't be trivially matched with glob, then don't use glob but instead provide valueFrom with an expression which constructs the path based on the inputs.

Use glob: * to match all files (recursively? maybe glob: **?) which can be filtered by an expression in valueFrom.

With that, we can continue to have the stdin and stdout fields on the tool object be either a basic string or an expression. We can eliminate ref in favor of a JsonPointer expression.

Thoughts?

from common-workflow-language.

Agreed, but if we're going that route, why not allow expressions in glob and do away with valueFrom completely? If glob pattern is/resolves to a file name, it will just match that file.

valueFrom seems useful when paired with loadContent to report something other than file, so perhaps worth keeping.

from common-workflow-language.

stdin/stdout flags rejected

glob permits expressions

secondaryFiles pattern matching implemented (more discussion about secondaryFiles in #42)

changed "valueFrom" to "outputEval"

from common-workflow-language.