Document avro-ld about common-workflow-language HOT 9 CLOSED

I would second this point. As far as I can tell, the schema description isn't usable by Avro.
I've taken the YAML https://github.com/common-workflow-language/common-workflow-language/blob/master/schemas/draft-3/cwl-avro.yml and converted to the JSON, then try to compile an interface with AVRO tools.

java -jar avro-tools-1.7.7.jar compile protocol cwl.json test_out
Gives the error:
Exception in thread "main" org.apache.avro.SchemaParseException: No protocol name specified:

And
java -jar avro-tools-1.7.7.jar compile schema cwl.json test_out
Gives the error:
Exception in thread "main" org.apache.avro.SchemaParseException: Type not supported: doc

Is there something I'm missing?

from common-workflow-language.

Try cwltool --print-avro

The schema language started as Avro but has increasingly diverged. The tools for working with it are in the cwltool/avro_ld module. I am working on spinning it off into a separate project with proper documentation, I hope to have something ready to push next week.

On August 4, 2015 1:38:22 PM EDT, Kyle Ellrott [email protected] wrote:

I would second this point. As far as I can tell, the schema description
isn't usable by Avro.
I've taken the YAML
https://github.com/common-workflow-language/common-workflow-language/blob/master/schemas/draft-3/cwl-avro.yml
and converted to the JSON, then try to compile an interface with AVRO
tools.

java -jar avro-tools-1.7.7.jar compile protocol cwl.json test_out
Gives the error:
Exception in thread "main" org.apache.avro.SchemaParseException: No
protocol name specified:

And
java -jar avro-tools-1.7.7.jar compile schema cwl.json test_out
Gives the error:
Exception in thread "main" org.apache.avro.SchemaParseException: Type
not supported: doc

Is there something I'm missing?

---

Reply to this email directly or view it on GitHub:
#69 (comment)

from common-workflow-language.

I tried

cwltool --print-avro > cwl.avro

Then

java -jar avro-tools-1.7.7.jar compile schema cwl.avro test

To which I received the error

Input files to compile:
  cwl.avro
Exception in thread "main" org.apache.avro.SchemaParseException: Can't redefine: SubworkflowFeatureRequirement
    at org.apache.avro.Schema$Names.put(Schema.java:1060)

from common-workflow-language.

In cwltool the schemas get fed into the Avro python module programmatically. I have not tried loading the exported schema into the java Avro implementation so I will have to look into it.

On August 6, 2015 2:28:01 AM EDT, Kyle Ellrott [email protected] wrote:

I tried

cwltool --print-avro > cwl.avro

Then

java -jar avro-tools-1.7.7.jar compile schema cwl.avro test

To which I received the error

Input files to compile:
 cwl.avro
Exception in thread "main" org.apache.avro.SchemaParseException: Can't
redefine: SubworkflowFeatureRequirement
  at org.apache.avro.Schema$Names.put(Schema.java:1060)

---

Reply to this email directly or view it on GitHub:
#69 (comment)

from common-workflow-language.

I thought the purpose of AVRO was to describe the serialization schema in such a way that code generation was possible.
What is AVRO-LD bringing to this project that real AVRO doesn't provide?

from common-workflow-language.

Yes, it makes same that you want to do code generation and that should be a supported use case.

Avro-ld provides four things:

Record inheritance: records can extend other records.

Template specialization: records can redefine the types of fields in the parent type.

Abstract types: a field can specify a base type and it will accept (and validate) any concrete subclass of that type.

Json-ld annotations: generate a json-ld context which can be used to interpret the CWL document as RDF.

Avro was adopted because the GA4GH initially adopted it for defining GA4GH schemas, but CWL ran up against Avro limitations which were addressed by creating avro-ld; it is worth noting that some GA4GH task teams are having similar problems.

On August 11, 2015 2:31:32 AM EDT, Kyle Ellrott [email protected] wrote:

I thought the purpose of AVRO was to describe the serialization schema
in such a way that code generation was possible.
What is AVRO-LD bringing to this project that real AVRO doesn't
provide?

---

Reply to this email directly or view it on GitHub:
#69 (comment)

from common-workflow-language.

(Cross-referenced with #111)

I've made an initial commit of the work to spin off the schema language into its own project:

https://github.com/common-workflow-language/schema_salad

Please note as of this writing it's not quite ready, there are a couple of issues that I still have to resolve, but I wanted to put the code out there to show it's pretty close.

In particular, it is able to export avsc schemas that are compatible with the Java Avro. I have successfully tested code generation.

Once I resolve the outstanding issues, the next step will be to convert cwltool and the draft-3 spec to use the new "Salad" schemas.

from common-workflow-language.

Schema Salad / CWL / Avro preview:

1. Check out master https://github.com/common-workflow-language/schema_salad
2. Check out salad_schema branch of CWL
3. cd schema_salad
4. python setup.py develop
5. python -mschema_salad --print-avro ../common-workflow-language/schemas/draft-3/cwl-avro.yml > cwl.avsc
6. java -jar avro-tools-1.7.7.jar compile schema cwl.avsc cwl

From there I've tried compiling the autogenerated code but haven't done anything with it, but this should be enough to get started. See issue #117 about the "Any" type. We still need to find a workaround.

Let me know how it goes.

refs #111 #116

from common-workflow-language.

Thank you @porterjamesj for your question. It looks to me like @tetron has documented avro-ld/schema_salad fairly well for now so I'm going to close this issue. Feel free to re-open it or make a new issue on the schema_salad repo if something else comes up.

from common-workflow-language.