rabix / cwl-ts Goto Github PK

Typescript data model for Common Workflow Language

License: Apache License 2.0

TypeScript 92.18% Shell 0.21% JavaScript 2.36% Common Workflow Language 5.26%

common-workflow-language workflow cwl typescript

cwl-ts's Introduction

cwl-ts

CWL-ts is a data model library for the Common Workflow Language made for use in TypeScript and JavaScript (primarily client-side) applications.

It supports JSON schemas and mappings for draft-3, draft-4, and v1.0. Draft-2 support is SevenBridges flavored.

Installation

npm install cwlts

Build

The npm package can be built locally by running the following command. It will generate .d.ts and compiled .js files into the /lib directory.

./scripts/package.sh

Mappings

lib/mappings contains TypeScript interfaces that can be used for type hinting when working with CWL documents in TS/JS.

Schemas

lib/schemas contains JSON Schema definitions for CommandLineTool, Workflow and ExpressionTool for each draft/version. They can be used with a JSON Schema validator to ensure schema validity of a CWL document.

Models [WIP]

lib/models contains TypeScript classes representing CWL entities. Models are still in early development, so many intended features are not implemented yet or don't function properly. The purpose of these models is to provide a set of methods that will facilitate working with CWL in a client-side application--one that graphically displays CWL documents or creates them. They should abstract differences between drafts and versions, provide a consistent API, and always generate correct and valid CWL documents.

CommandLineToolModel should:

generate command line when supplied test data
evaluate and validate expressions
enable operations on complex child objects (inputs, outputs, arguments, etc)

WorkflowModel should:

generate DAG of workflow
enable adding and connecting steps
enable operations on complex child objects (inputs, outputs, steps, etc)

cwl-ts's People

Contributors

Stargazers

Watchers

Forkers

mayacoda matebordas ivanbatic arvados stain datirium yoanncouillec amaris bogdang989 mr-c michael-kotliar wuqunfei gungorbudak namkilee jdu86 goelnandan 382982408

cwl-ts's Issues

Generate command line for v1.0 tools

sbg:draft-2 tools can generate their own command line by passing baseCommand, inputs, arguments and stdin/stdout through a series of parsers which resolve each chunk recursively and return a CommandLinePart

How this works can be found in SBDraft2CommandLineToolModel.ts:185:

private generateCommandLineParts(): Promise<CommandLinePart[]> {
    // creates a flat list of inputs and arguments, sorting them accordingly in the process

    // creates a flat list of job values (flattens record fields, for example)
   
    // runs baseCommands through the parser, returns array of Promises

    // runs flattened inputs and arguments through the parser

    // runs streams through the parser

    // joins and returns all Promise arrays, filtering null promises (if job value was null)
}

The same needs to be done with v1.0 tools, keeping in mind any differences in command lines between draft-2 and v1.0.

Also, I'm sure there are bugs in the current command line, for example I don't think some cases where valueFrom is defined are covered properly.

Missing dependency in npm package

I get the following message when running ng serve on a project which uses cwl-ts:

ERROR in ./node_modules/cwlts/models/helpers/JSExecutor.js
Module not found: Error: Can't resolve 'vm' in './node_modules\cwlts\models\helpers'

Running npm install vm solves this error.
I think that the vm module needs to be added to package.json

Problem with valueFrom field

rabix/composer#472

Constructs like

    valueFrom: O=$(runtime.outdir)/rna_metrics.txt

result in instability - the command line preview will show errors, which disappear and then reappear with edits in the code pane.

@gungorbudak has tracked these to the latest cwl-ts release

Missing name field for SchemaDefRequirement types (CWL issue)

mayacoda commented 17 hours ago
SchemaDefRequirement types (InputRecordSchema, InputArraySchema, and InputEnumSchema) should have the name field. It exists in the schema-def example, but not in the documentation.

common-workflow-language/common-workflow-language#216

Map the Schema Salad draft-3

More info on http://www.commonwl.org/draft-3/SchemaSalad.html

Validation for Workflows

Package only compiled .js and .d.ts files in node module

Commit compiled code and package that.

Add timeout on js execution

Expression execution is susceptible to infinite loops. The JSExecutor used by default doesn't timeout, even though there is a timeout set in the options. This might be a bug caused by the nested Promises used to make execution asynchronous.

JSExecutor.ts

 static evaluate(version: CWLVersion, expr: string, job?: any, self?: any): Promise<any> {
        const options = {
            displayErrors: true,
            timeout: 1000
        };
       ...
}

build package leads to typescript compiler error

Hi all!

I try to compile my own version of CWL-TS.

I followed readme instructions by running scripts/package.sh.
However it raises many typescript compiler errors.

Has someone been able to build CWL-TS locally ?

Thanks!

Implementing the data model/architecture approach

CWL v1.0

CWL v1.0 has now been released! 🎉

Draft-4 should be renamed to v1
Mappings should reflect new features

V1.0 schemas do not support arrays as maps

CWL v1.0 supports lists being written as maps (common-workflow-language/common-workflow-language#237). JSON schemas for v1.0 don't support this behavior.

Add steps on the fly

cwl-ts/src/models/generic/WorkflowModel.ts

Line 1077 in c8355c5

protected addStepToGraph(step: StepModel, graph: Graph = this.graph) {

Would it be possible to make this method public ?
Then it would be possible to create steps on the fly.

Error: Can't resolve 'vm'

I created a simple Angular application and imported cwlts/models. When the application is compiled it complains about 'vm' module not present.

It is fixed if I install 'vm' (npm install vm).

Maybe you should add the 'vm' module to the list of dependencies in your package.json.

ERROR in ./node_modules/cwlts/models/helpers/JSExecutor.js
Module not found: Error: Can't resolve 'vm' in '/Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/models/helpers'
resolve 'vm' in '/Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/models/helpers'
  Parsed request is a module
  using description file: /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/package.json (relative path: ./models/helpers)
    Field 'browser' doesn't contain a valid alias configuration
    resolve as module
      looking for modules in /Users/eduardo/Development/glue/cwlts-angular/
        using description file: /Users/eduardo/Development/glue/cwlts-angular/package.json (relative path: .)
          Field 'browser' doesn't contain a valid alias configuration
          using description file: /Users/eduardo/Development/glue/cwlts-angular/package.json (relative path: ./vm)
            no extension
              Field 'browser' doesn't contain a valid alias configuration
              /Users/eduardo/Development/glue/cwlts-angular/vm doesn't exist
            .ts
              Field 'browser' doesn't contain a valid alias configuration
              /Users/eduardo/Development/glue/cwlts-angular/vm.ts doesn't exist
            .js
              Field 'browser' doesn't contain a valid alias configuration
              /Users/eduardo/Development/glue/cwlts-angular/vm.js doesn't exist
            as directory
              /Users/eduardo/Development/glue/cwlts-angular/vm doesn't exist
      /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/models/helpers/node_modules doesn't exist or is not a directory
      /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/models/node_modules doesn't exist or is not a directory
      /Users/eduardo/Development/glue/cwlts-angular/node_modules/node_modules doesn't exist or is not a directory
      /Users/eduardo/Development/glue/node_modules doesn't exist or is not a directory
      /Users/eduardo/Development/node_modules doesn't exist or is not a directory
      /Users/eduardo/node_modules doesn't exist or is not a directory
      /Users/node_modules doesn't exist or is not a directory
      /node_modules doesn't exist or is not a directory
      looking for modules in /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/node_modules
        using description file: /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/package.json (relative path: ./node_modules)
          Field 'browser' doesn't contain a valid alias configuration
      looking for modules in /Users/eduardo/Development/glue/cwlts-angular/node_modules
        using description file: /Users/eduardo/Development/glue/cwlts-angular/package.json (relative path: ./node_modules)
          Field 'browser' doesn't contain a valid alias configuration
          using description file: /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/package.json (relative path: ./node_modules/vm)
            no extension
              Field 'browser' doesn't contain a valid alias configuration
          using description file: /Users/eduardo/Development/glue/cwlts-angular/package.json (relative path: ./node_modules/vm)
            no extension
              Field 'browser' doesn't contain a valid alias configuration
              /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/node_modules/vm doesn't exist
            .ts
              Field 'browser' doesn't contain a valid alias configuration
              /Users/eduardo/Development/glue/cwlts-angular/node_modules/vm doesn't exist
            .ts
              Field 'browser' doesn't contain a valid alias configuration
              /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/node_modules/vm.ts doesn't exist
            .js
              Field 'browser' doesn't contain a valid alias configuration
              /Users/eduardo/Development/glue/cwlts-angular/node_modules/vm.ts doesn't exist
            .js
              Field 'browser' doesn't contain a valid alias configuration
              /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/node_modules/vm.js doesn't exist
              /Users/eduardo/Development/glue/cwlts-angular/node_modules/vm.js doesn't exist
            as directory
              /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/node_modules/vm doesn't exist
            as directory
              /Users/eduardo/Development/glue/cwlts-angular/node_modules/vm doesn't exist
[/Users/eduardo/Development/glue/cwlts-angular/vm]
[/Users/eduardo/Development/glue/cwlts-angular/vm.ts]
[/Users/eduardo/Development/glue/cwlts-angular/vm.js]
[/Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/models/helpers/node_modules]
[/Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/models/node_modules]
[/Users/eduardo/Development/glue/cwlts-angular/node_modules/node_modules]
[/Users/eduardo/Development/glue/node_modules]
[/Users/eduardo/Development/node_modules]
[/Users/eduardo/node_modules]
[/Users/node_modules]
[/node_modules]
[/Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/node_modules/vm]
[/Users/eduardo/Development/glue/cwlts-angular/node_modules/vm]
[/Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/node_modules/vm.ts]
[/Users/eduardo/Development/glue/cwlts-angular/node_modules/vm.ts]
[/Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/node_modules/vm.js]
[/Users/eduardo/Development/glue/cwlts-angular/node_modules/vm.js]
 @ ./node_modules/cwlts/models/helpers/JSExecutor.js 3:9-22
 @ ./node_modules/cwlts/models/helpers/index.js
 @ ./node_modules/cwlts/models/index.js
 @ ./src/app/app.component.ts
 @ ./src/app/app.module.ts
 @ ./src/main.ts
 @ multi ./src/main.ts

Command line generation

Either as a separate parser or as part of the model, sample command lines should be generated from CLT descriptions.

SecondaryFiles, streamable and format should not be assignable to type record and enum (CWL issue)

This is an issue with the CWL (common-workflow-language/common-workflow-language#215) but we can fix it in the typings.

The draft-3 specification states that CommandInputRecordSchema, CommandInputEnumSchema, InputRecordSchema and InputEnumSchema have the fields secondaryFiles, streamable and format. These fields are only possible for type: File or items: File which types record and enum are not. They should be removed from the specification.

use schema.org for license, copyright holder

https://github.com/ProteinsWebTeam/ebi-metagenomics-cwl/blob/master/tools/esl-sfetch-index.cwl#L44

Prevent cwl-ts from deleting requirements

cwl-ts deletes workflow requirements and then puts them back. This should be fixed such that cwl-ts only adds requirements, never deletes existing ones.

Validation for Command Line Tools

Write a descriptive readme and contribution guidelines

Question: does it support v1.2.0-dev4 or v1.2.0 version?

Just curious if this package can properly parse CWL files with cwlVersion: v1.2.0-dev4 or cwlVersion: v1.2.0

Thanks

serialize to array of array instead of array

cwl-ts/src/models/d2sb/SBDraft2WorkflowModel.ts

Line 165 in c8355c5

 base.inputs = <SBGWorkflowInputParameter[]> this.inputs.map(i => i.serialize()); 

Hi!
I faced a little bug while serializing CWL workflow.
When inputs type array contains more than one type it is serialize to an array of array like:
[ [ "int", "File" ] ]
However when the model contains only one typem the serialization is ok:
[ "File" ]

Create interfaces for the Workflow draft-3 specification

We need to create typings according to http://www.commonwl.org/draft-3/Workflow.html

Implement reference resolution

This task includes $import and $include directives.

Analyse similar data structures and abstract away their properties in CL tool mapping

Example creating workflow programmatically

Please, could you provide an example of how to create a workflow from scratch?

It would be nice if it contained some of these:

Add workflow's input and output
Add a couple of steps with its inputs and outputs
Edit label and doc
Serialize

Thanks!

Implement typings for draft-2 and draft-4

Construct a graph from a document containing an array of workflows

I would appreciate advice on how to create a workflow model from a document that contains an array of workflows.

There is an example below of this type of document.

When I create the model I can see the customProps.$graph populated. I would like to construct a graph using #main as the entry point.

Thanks for your help!

{
    "cwlVersion": "v1.0", 
    "$graph": [
        {
            "class": "CommandLineTool", 
            "baseCommand": "dbfetch_lwp.pl", 
            "hints": [
                {
                    "dockerPull": "psafont/ebitools:20170602", 
                    "class": "DockerRequirement"
                }
            ], 
            "inputs": [
                {
                    "type": "File", 
                    "doc": "List of accessions to be retrieved as sequences.", 
                    "inputBinding": {
                        "loadContents": true, 
                        "valueFrom": "$(self.contents)", 
                        "position": 3
                    }, 
                    "id": "#dbfetch.cwl/accessions"
                }, 
                {
                    "type": "string", 
                    "doc": "Database to be searched.", 
                    "inputBinding": {
                        "position": 2
                    }, 
                    "default": "uniprot", 
                    "id": "#dbfetch.cwl/database"
                }, 
                {
                    "type": "string", 
                    "doc": "Type of command-line interface.", 
                    "inputBinding": {
                        "position": 1
                    }, 
                    "default": "fetchBatch", 
                    "id": "#dbfetch.cwl/method"
                }, 
                {
                    "type": "string", 
                    "doc": "Format of the output", 
                    "inputBinding": {
                        "position": 4
                    }, 
                    "default": "fasta", 
                    "id": "#dbfetch.cwl/outformat"
                }, 
                {
                    "type": "string", 
                    "doc": "Style of the output", 
                    "inputBinding": {
                        "position": 5
                    }, 
                    "default": "raw", 
                    "id": "#dbfetch.cwl/outstyle"
                }
            ], 
            "outputs": [
                {
                    "type": "stdout", 
                    "id": "#dbfetch.cwl/sequences"
                }
            ], 
            "id": "#dbfetch.cwl"
        }, 
        {
            "class": "CommandLineTool", 
            "baseCommand": "sh", 
            "hints": [
                {
                    "dockerPull": "psafont/ebitools:20170602", 
                    "class": "DockerRequirement"
                }
            ], 
            "inputs": [
                {
                    "type": "File", 
                    "doc": "File with proteins accessions whose sequences are to be retrieved.", 
                    "inputBinding": {
                        "position": 1
                    }, 
                    "id": "#idsFordbfetch.cwl/accessions_file"
                }, 
                {
                    "type": "string", 
                    "inputBinding": {
                        "position": 0
                    }, 
                    "default": "/usr/src/ebitools/idsFordbfetch.sh", 
                    "id": "#idsFordbfetch.cwl/command"
                }, 
                {
                    "type": "string", 
                    "doc": "Maximum number of records to fetch", 
                    "inputBinding": {
                        "position": 2
                    }, 
                    "default": "20", 
                    "id": "#idsFordbfetch.cwl/numberAccessions"
                }
            ], 
            "outputs": [
                {
                    "type": "stdout", 
                    "id": "#idsFordbfetch.cwl/accessions"
                }
            ], 
            "id": "#idsFordbfetch.cwl"
        }, 
        {
            "label": "Retrieve biological records", 
            "doc": "Retrieve biological records by accession identifier", 
            "class": "Workflow", 
            "inputs": [
                {
                    "label": "File with identifiers", 
                    "doc": "File with identifiers", 
                    "type": "File", 
                    "id": "#main/accessions"
                }, 
                {
                    "label": "Number of records to retrieve", 
                    "doc": "Number of records to retrieve starting from the top.", 
                    "type": [
                        "null", 
                        "string"
                    ], 
                    "id": "#main/numberAccessions"
                }
            ], 
            "outputs": [
                {
                    "label": "Biological records", 
                    "doc": "Output file with retrieved biological records", 
                    "type": "File", 
                    "outputSource": "#main/fetch/sequences", 
                    "id": "#main/sequences"
                }
            ], 
            "steps": [
                {
                    "label": "Dbfetch", 
                    "doc": "Retrieve biological records from identifiers", 
                    "run": "#dbfetch.cwl", 
                    "in": [
                        {
                            "source": "#main/format/accessions", 
                            "id": "#main/fetch/accessions"
                        }
                    ], 
                    "out": [
                        "#main/fetch/sequences"
                    ], 
                    "id": "#main/fetch"
                }, 
                {
                    "label": "Get X top identifiers", 
                    "doc": "Filter the X top identiers", 
                    "run": "#idsFordbfetch.cwl", 
                    "in": [
                        {
                            "source": "#main/accessions", 
                            "id": "#main/format/accessions_file"
                        }
                    ], 
                    "out": [
                        "#main/format/accessions"
                    ], 
                    "id": "#main/format"
                }
            ], 
            "id": "#main"
        }
    ]
}