CI should run conformance tests and the extension test suite.

CWL import/include directives should be resolved before execution.

• Assure workflow is a connected DAG
• Check required inputs of master job

• Backends should register with AMQ configs.
• Worker should also be extracted as backend
• SimpleRabixTool backend example

POST to /backends with AMQ JSON config (send/receive/heartbeat queues) to create backend

fixed it

When specifying input files through the job JSON file, relative paths should be relative to the location of the inputs JSON file.

When specifying paths directly on the command line, they should be relative to the current working directory.

In service mode, jobs should be persisted to disk.

For draft-2, Workflow.steps[0].inputs[0].default gets ignored.

I don't think it is correctly pulling the image down and is just running on local system.

Hi there,

it is possible to have the release packaged into a folder with the version number?

So instead of

examples/
examples/dna2protein/
examples/dna2protein/data/
config/
config/development/
config/producetion/
config/test/
examples/README.md
examples/dna2protein/inputs.json
examples/dna2protein/transcribe.cwl.json
examples/dna2protein/dna2protein.cwl.json
examples/dna2protein/README.md
examples/dna2protein/translate.cwl.json
examples/dna2protein/data/input.txt
config/core.properties
config/ftp.properties
config/logback.xml
config/logback-verbose.xml
lib/rabix-backend-local-0.0.1-SNAPSHOT.jar
rabix

something like

rabix-0.6.0-alpha/*

and it is also possible to pack the rabix command line as executable?

What do you think?

Worker should add checksum and size properties in post-process phase. This should also apply to file literals (not necessarily made by expression tools).

Default files referenced from the workflow seem to be found relative to the input file rather then the workflow file. The reference implementation does to opposite. Because the workflows and input files are in the same folder in the conformance test, this behavior probably isn't noticed in testing.

Example workflow (in the same directory put a README.md file)

class: CommandLineTool
cwlVersion: v1.0
hints:
  DockerRequirement:
    dockerPull: ubuntu

baseCommand: [md5sum]
inputs:
  script:
    type: File
    default:
      class: File
      location: README.md
    inputBinding:
      position: 1

stdout: out.txt

outputs:
  checksum:
    type: File
    outputBinding:
      glob: out.txt

Create an empty dictionary file named empty_input.json
If empty_input.json is in the same directory as the workflow, the command
java -jar rabix/lib/rabix-backend-local.jar default_file.cwl empty_input.json will work.
But if it is move to tmp/empty_input.json then the command java -jar rabix/lib/rabix-backend-local.jar default_file.cwl tmp/empty_input.json will fail because

$ cat tmp/4983a396-9289-41c9-996b-9c0f4c8e516b/root/job.err.log
md5sum: /Users/ellrott/workspaces/gaea-bmeg/tmp/README.md: No such file or directory

It's not possible to symlink the binary right now:

mattions@apollo:bin$ ls -l
total 4020
lrwxrwxrwx 1 mattions mattions      48 Dec 12 13:54 rabix -> /home/mattions/SBG/Progs/rabix-0.7.0-alpha/rabix

mattions@apollo:bin$ rabix
Error: Unable to access jarfile /home/mattions/bin/lib/rabix-backend-local.jar

Hi,
Just confused by bunny project. My understranding rabix is implemented in python for CWL specification. Why there is a bunny of java version under rabix? What's the value of bunny while there is already a more matured python version. It looks to me that python version should be continued and now almost dead. Will python version be deprecated and replaced by bunny?

Example:

./rabix testbacklog/inputs/grep.cwl.yaml testbacklog/grep.inputs.yaml

Where tool : testbacklog/inputs/grep.cwl.yaml is an empty file.

Based on the Task Execution Schema spec by GA4GH.

Leftover items for initial version:

• Include in build (backend.tes.url config var)
• Remaining conformance tests (10 failing atm)
• Make a project on ci.commonwl.org

Create second bindings/protocol implementation "SimpleRabixTool" to demonstrate multiple protocols.

These should be simple command line templates that do not declare input ports and have a single output port "stdout" which reports a file.

Example:

#rabix:SimpleRabixTool
grep $(pattern) $(infile) > stdout.txt

Some missing features:

• Link merge method
• Scatter method

would this work?

"fileDef":
{
"fileContent": {
"class": "Expression",
"script": "'#!/bin/bash \n\\n#/samtools/samtools view -H '+$job.inputs.bam.path+' > header.sam \n\\n\n\\n#filter for q20 \n\\n/samtools/samtools view -q 20 -b '+$job.inputs.bam.path+' > filtered.bam \n\\n\n\\n#cat header.sam filtered.bam > result.sam';\n",
"engine": "#cwl-js-engine"
},
"filename": "filter_for_q20.sh"
}

In dna2protein.cwl.json file under example data folder, there is empty baseCmd item, this is not defined in CWL schema, should this be removed? @gaurav-kaushik

We need to introduce common format in order to support multiple protocols inside the same payload.

Hi,

New docker for Mac uses OS X Hypervisor.framework for virtualization, instead of external virtualization programs.

I tried bunny 0.6.4 with it and it gets stuck on pulling an image (manually pulling from shell works). Docker for Mac exposes the docker API on a socket in /var/tmp/docker.sock.

rabix/rabix#145

Debug logs should only be printed with -v flag. Otherwise, info messages should go to stderr with messages for each step:

• Started
• Pulling docker image
• Running command line
• Finished

Also, usage string and error messages should go to stderr.

Needed for conformance test 70.

These methods have to be moved to helper.

Set getInputFiles(Job job) throws BindingException;
Set getOutputFiles(Job job, boolean onlyVisiblePorts) throws BindingException; // TODO remove onlyVisiblePorts
Job updateInputFiles(Job job, FileTransformer fileTransformer) throws BindingException;
Job updateOutputFiles(Job job, FileTransformer fileTransformer) throws BindingException;
Map<String, Object> translateFile(FileValue fileValue);
DataType getDataTypeFromValue(Object input);
List getFilesFromValue(Object input);
Object updateFileValues(Object input, FileTransformer fileTransformer)

These two methods have to be changed to use common file/directory values.

Job mapInputFilePaths(Job job, FilePathMapper fileMapper) throws BindingException;
Job mapOutputFilePaths(Job job, FilePathMapper fileMapper) throws BindingException;

On SBG platform we have Decompressor application which extracts files from the input archive to decompressed_files directory. The App has the following glob: decompressed_files/!(*.meta)
which seems that is not supported in rabix-0.6.6-alpha release. cwl.output.json is empty when files in decompressed_files directory exist.

• sbg:stageInput
• sbg:inheritMetadata
• sbg:metadata
• CPU and memory hints

Hi, trying to run the example from the README, running into an issue here:

rjhb451:rabix spanglry$ ./rabix examples/dna2protein/dna2protein.cwl.json examples/dna2protein/inputs.json
[2016-12-01 15:25:22.280] Pulling docker image rfranklin/pythondev

It gets to this point and hangs indefinitely (I have had it running in the background while I do other stuff, up to half an hour now). I thought this might be my issue, but I can pull and run other docker images fine.

Any thoughts? Thanks!

./rabix.sh -a example/grep-app.json -i example/grep-inputs.json is not valid anymore.

Only needed for conformance tests.

Should be able to run locally on OSX if docker toolbox is installed.

Include TES backend into the local Bunny executor #75 . Add tes-url command line argument.

Not the highest priority but tools that have no input ports might be causing an issue. I have a tool which simply creates a .csv file which is input to another tool. Works on SBG platform but maybe not locally with bunny.

Issue is solved by putting the csv files in the second tool's wrapper.

Thanks

Dave