clean up top-level site navigation (for docs and for code) about ga4gh-schemas HOT 40 CLOSED

A few more comments from the discussion group:

• the top-level doc URL (ga4gh.github.io) isn't very memorable
• it's not obvious that both code and doc exist (which leads people looking for doc to get stuck in and overwhelmed by code)
• the APIs link at the top of the doc page does the right thing, but is easy to miss (given how quiet it is compared to the louder text in the body, that takes you to individual repos)

from ga4gh-schemas.

I don't understand why we don't have just three directories, a top one with documents relevant to the whole Data working group,
and a subdirectory for each task group with documents relevant to that task group: the Avro and the documentation. I would only
make a subdirectory beyond those once there were more than 10-20 documents in a natural subclass.

That much flatter structure would look natural to most people I think.

Apologies for my luddite comments. Personally I managed to find the APIs but have never found the documentation, though I haven't looked hard.

Richard

On 8 May 2014, at 14:47, David Glazer [email protected] wrote:

A few more comments from the discussion group:

the top-level doc URL (ga4gh.github.io) isn't very memorable
it's not obvious that both code and doc exist (which leads people looking for doc to get stuck in and overwhelmed by code)
the APIs link at the top of the doc page does the right thing, but is easy to miss (given how quiet it is compared to the louder text in the body, that takes you to individual repos)
—
Reply to this email directly or view it on GitHub.

The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.

from ga4gh-schemas.

@richarddurbin , I don't love the extra clicks to get to the Avro, but I do like that we have a structure that is ready for growth -- we should be able to add in sample apps and reference implementations, with all the associated auto-build machinery, without moving stuff around. Open to ideas on how to get both, but I don't have any.

(And btw -- thanks to @massie and @cassiedoll for setting up what we have -- I think it's already working well for contributors, and only a few small changes away from working well for casual visitors also.)

from ga4gh-schemas.

There are a few problems with the current structures:

1. It is hard to find the relevant information unless you are very familiar with it.
2. Task groups are disconnected. For example, Benedict wanted to include ga4gh.avdl, but it is in a separate git repository and cannot be referenced. For another example, there is a beacon.avdl in ReadTaskGroup, but there is a separate Beacon repository at the top level. After all, various task groups will be connected (VarRef task group is connecting to ReadTaskGroup). It makes more sense to put them in one repository by merging the others into ReadTaskGroup and then renaming the repository name.
3. File naming is not appropriate because task groups are separated. For example, ga4gh.avdl in ReadTaskGroup is not quite right. It primarily describes the read store only.

from ga4gh-schemas.

Helo David and everyone,

I can David's points are good. Thanks also to @massie and @cassiedoll.
Sorry, my previous email was unnecessarily intemperate.
Maybe as an alternative we can just put direct links on the main text of the front page to the currently active avro idl specification pages and the top level documents, including "how to"?
As things develop with implementations etc. the links on the top page can change. This would make the top page more of a current menu to go to the current activities of the working group.

Richard

On 8 May 2014, at 15:02, David Glazer [email protected] wrote:

@richarddurbin , I don't love the extra clicks to get to the Avro, but I do like that we have a structure that is ready for growth -- we should be able to add in sample apps and reference implementations, with all the associated auto-build machinery, without moving stuff around. Open to ideas on how to get both, but I don't have any.

(And btw -- thanks to @massie and @cassiedoll for setting up what we have -- I think it's already working well for contributors, and only a few small changes away from working well for casual visitors also.)

—
Reply to this email directly or view it on GitHub.

The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.

from ga4gh-schemas.

RE:

Task groups are disconnected. For example, Benedict wanted to include ga4gh.avdl, but it is in a separate git repository and cannot be referenced. For another example, there is a beacon.avdl in ReadTaskGroup, but there is a separate Beacon repository at the top level. After all, various task groups will be connected (VarRef task group is connecting to ReadTaskGroup). It makes more sense to put them in one repository by merging the others into ReadTaskGroup and then renaming the repository name.

I don't think the correct solution is to have one large repository. I think the preferable approach is to publish our artifacts (e.g., to maven), so that other teams can pull the artifacts from wherever they are published. It will be necessary to publish artifacts later anyways, so that external projects (Company X's implementation of the GA4GH reads API, etc) have a clear way to pull the schemas.

from ga4gh-schemas.

Unfortunately, for the use case of referencing .avdl schemas in one project
from another, the standard Maven way of pulling dependency jars won't cut
it. The dependent project needs the dependency's source for the .avdl file
in order to build, and it needs to be available at a predictable relative
filesystem path for the compiler to find it.

The easiest way to set that up would be via Git submodules, but that just
copies the one project inside the other. If we went this route, I would
suggest splitting the .avdl files into their own separate Git repository.

A more engineering-intensive way would be to implement the method described
here: <
http://mail-archives.apache.org/mod_mbox/avro-user/201212.mbox/%3C97CDF5378BDCEA4FB609A50F0FD3599A833A9AF8@TUK2-EMSMBX3.intelius1.intelius.com%3E

Basically, the dependency (this project) needs to be modified as described
at <Mhttp://
mail-archives.apache.org/mod_mbox/avro-user/201212.mbox/%3C97CDF5378BDCEA4FB609A50F0FD3599A833A8A29@TUK2-EMSMBX3.intelius1.intelius.com%3E>
in order to build and publish a source .jar containing just the Avro
schemas. Then the dependent project gets some more Maven wizardry to pull
down and extract that source .jar into a predictable place so that schemas
can reference it.

I think the second approach is probably better, even though it requires a
bunch of engineering in the dependent project. I can make an issue (or
maybe a pull request?) for that in a bit.

On Thu, May 8, 2014 at 8:14 AM, Frank Austin Nothaft <
[email protected]> wrote:

RE:

Task groups are disconnected. For example, Benedict wanted to include
ga4gh.avdl, but it is in a separate git repository and cannot be
referenced. For another example, there is a beacon.avdl in ReadTaskGroup,
but there is a separate Beacon repository at the top level. After all,
various task groups will be connected (VarRef task group is connecting to
ReadTaskGroup). It makes more sense to put them in one repository by
merging the others into ReadTaskGroup and then renaming the repository name.

I don't think the correct solution is to have one large repository. I
think the preferable approach is to publish our artifacts (e.g., to maven),
so that other teams can pull the artifacts from wherever they are
published. It will be necessary to publish artifacts later anyways, so that
external projects (Company X's implementation of the GA4GH reads API, etc)
have a clear way to pull the schemas.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40#issuecomment-42562258
.

from ga4gh-schemas.

Hmm.. this maven thing looks overcomplicated to me while make it harder to access and integrate schemas. What is the benefit of separating ReadTaskGroup, VarRef task group, beacon etc. into separate repositories? We can still achieve auto-build after merging them together. I am not sure what "ready for growth" means. Does it mean we will put sample apps and reference implementations inside ReadTaskGroup? I would rather put all schemas in one git repository, and put each app/implementation in a separate repository having the schema repository as a git submodule.

from ga4gh-schemas.

@lh3 Maven is generally not complicated, but as @adamnovak points out, it may not be a good decision for our specific case. I don't think anyone is suggesting that we put sample apps/reference implementations into the ReadTaskGroup repository/etc.; my implication is that we'll want to publish artifacts for public consumption at a later date, which is a somewhat orthogonal concern.

I'll let @massie chime in, as he may know a workaround for the issues @adamnovak points out. If a good workaround doesn't exist, then I agree that having a unified repository for all schemas is a good approach.

from ga4gh-schemas.

@fnothaft Thanks for the reply. I agree we can wait to see if @massie has a cleaner solution.

Meanwhile, I am still curious about the benefit of separating task teams into separate repositories, and what @dglazer means by "ready for growth".

from ga4gh-schemas.

I thought the reason for the separate repositories was that we were going
to produce two different APIs (or collections of APIs): one for people
working with read data, and one for people working with variant data. You
might want to pull in the variant API but not the read API if you're doing
a genome-wide association study, for example. Or in some applications you
would use both.

On Thu, May 8, 2014 at 11:30 AM, Heng Li [email protected] wrote:

@fnothaft https://github.com/fnothaft Thanks for the reply. I agree we
can wait to see if @massie https://github.com/massie has a cleaner
solution.

Meanwhile, I am still curious about the benefit of separating task teams
into separate repositories, and what @dglazer https://github.com/dglazermeans by "ready for growth".

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40#issuecomment-42587072
.

from ga4gh-schemas.

Thanks for the clarification, @adamnovak. When our programs call a library, we are open to all its components even if we don't use them. I think the same can be applied in our case. If someone want to develop APIs, they can pull all the schemas but only use the part of interest. Our schemas are small and unlikely to cause bloated code. With schemas put together, it is easier for a user and us as well to find them, to understand their relationships, to improve info exchanges and to avoid inconsistencies (e.g. naming conflict).

from ga4gh-schemas.

While we are at it, can somebody please put under

https://github.com/ga4gh/FileFormatsTaskTeam

a link to the file formats task team GitHub site

https://github.com/samtools/hts-specs

Thanks!!! -D

On Thu, May 8, 2014 at 1:56 PM, Heng Li [email protected] wrote:

Thanks for the clarification, @adamnovak https://github.com/adamnovak.
When our programs call a library, we are open to all its components even if
we don't use them. I think the same can be applied in our case. If someone
want to develop APIs, they can pull all the schemas but only use the part
of interest. Our schemas are small and unlikely to cause bloated code. With
schemas put together, it is easier for a user and us as well to find them,
to understand their relationships, to improve info exchanges and to avoid
inconsistencies (e.g. naming conflict).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40#issuecomment-42605167
.

from ga4gh-schemas.

While we are at it, can somebody please put under [FileFormatsTaskTeam] a link to [hts-specs]

Good point -- I've put some basic placeholder text there now.

from ga4gh-schemas.

• 1. Re @lh3's question:

Meanwhile, I am still curious about ... what @dglazer means by "ready for growth".

I've been assuming that we'll eventually want to include actual executable code (e.g. a reference implementation of a server, client libraries to make it easy to call the API in different environments, sample apps such as a genome browser). But that's an assumption; curious if others agree.

• 2. Re directory layout for all our AVDL files -- I'm open to a unified directory, and would love to hear from @massie and others who have worked with these tools before. I do agree that it's useful to think of a single GA4GH Genomics API, with different subsets (e.g. read methods, variant methods) that can be independently implemented but work well together.
• 3. Re basic nav cleanup (the original topic) -- we should do that in all cases. Volunteers welcome; I may take a crack at it this weekend, but please don't wait for me. (And don't laugh at my design sensibilities if I do :)

from ga4gh-schemas.

I submitted a baby step at nav cleanup in the ga4gh.github.io repo -- ptal.

from ga4gh-schemas.

I've been assuming that we'll eventually want to include actual executable code (e.g. a reference implementation of a server, client libraries to make it easy to call the API in different environments, sample apps such as a genome browser). But that's an assumption; curious if others agree.

I would prefer not to put schemas and implementations in the same repository. Implementations may frequently use schemas from more than one teams. ReadTaskTeam is likely to use schemas from metaTeam. VarRefTeam is calling ReadTaskTeam schema right now but doesn't know how to achieve that (#41). Having schemas together easily solves the problem with no negative effects so far as I am aware. The only issue is that we need to move files around, but we'd better do this earlier than later. What else prevents us from centralizing schemas?

I also reiterate the issue of schema inconsistency. In ReadTaskTeam schema, we use contigName and position, but in Beacon, we use chromosome and coordinate and have a new concept of referenceVersion. In ReadTaskTeam, we are thinking to adopt accession numbers in some way, but we are not talking about this in Beacon. Putting schemas together will make us more aware of such inconsistencies.

from ga4gh-schemas.

I am +1 to @lh3's comments about separating schemas and implementations. FWIW, in our work at Berkeley, we've found it preferable to both separate the schema and implementation, and to have the schemas for different data types (e.g., reads, variants, references) in the same repository, as there is significant overlap between data types.

from ga4gh-schemas.

@lh3 and @fnothaft -- the idea of merging schema files into one directory / repo is growing on me, for all the reasons you cite. It will make it a little trickier to coordinate edits from many different task teams who aren't talking regularly, but as @lh3 points out we're going to have to pay that coordination price at some point anyway. I'd still really like to hear from @massie on the thinking behind his initial setup, but unless he has new arguments, I'd be +1 for a change that put all the schemas in one place.

@lh3, re schema inconsistency -- agreed. I submitted #43 as a baby step (on file naming); please +1 it if you agree.

from ga4gh-schemas.

It will make it a little trickier to coordinate edits from many different task teams who aren't talking regularly

I think this should be fine as long as different task teams typically modify a few schema files relevant to their own team. This way commits from different task teams won't cause conflicts. We might also encourage to tag issues specific to each task team to alleviate the problem of a long and mixed issue list. EDIT: for example, put a ReadTaskTeam tag on an issue to indicate that the issue is primarily related to ReadTaskTeam.

from ga4gh-schemas.

@dglazer realistically, if we put all the schemas in a single repository, it forces coordination. It's better to pay the price up front, instead of diverging and needing to force convergence/coordination later.

from ga4gh-schemas.

+1 for putting all schemas in a single repository.

On Sat, May 10, 2014 at 9:24 AM, Frank Austin Nothaft <
[email protected]> wrote:

@dglazer https://github.com/dglazer realistically, if we put all the
schemas in a single repository, it forces coordination. It's better to pay
the price up front, instead of diverging and needing to force
convergence/coordination later.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40#issuecomment-42746496
.

from ga4gh-schemas.

I'm ccing people from other task teams and driver projects. I hope
everybody is on board with putting all our schemas in a single repository.

On Sat, May 10, 2014 at 10:35 AM, David Haussler [email protected]:

+1 for putting all schemas in a single repository.

On Sat, May 10, 2014 at 9:24 AM, Frank Austin Nothaft <
[email protected]> wrote:

@dglazer https://github.com/dglazer realistically, if we put all the
schemas in a single repository, it forces coordination. It's better to pay
the price up front, instead of diverging and needing to force
convergence/coordination later.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40#issuecomment-42746496
.

from ga4gh-schemas.

+1 for putting all schemas in a single repository.

On 10 May 2014, at 18:39, haussler [email protected] wrote:

I'm ccing people from other task teams and driver projects. I hope
everybody is on board with putting all our schemas in a single repository.

On Sat, May 10, 2014 at 10:35 AM, David Haussler [email protected]:

+1 for putting all schemas in a single repository.

On Sat, May 10, 2014 at 9:24 AM, Frank Austin Nothaft <
[email protected]> wrote:

@dglazer https://github.com/dglazer realistically, if we put all the
schemas in a single repository, it forces coordination. It's better to pay
the price up front, instead of diverging and needing to force
convergence/coordination later.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40#issuecomment-42746496
.

—
Reply to this email directly or view it on GitHub.

The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.

from ga4gh-schemas.

It looks like everyone is in agreement here on ending up with:

• one "schemas" repo in ga4gh which will contain all the avdl files.
• we'll use labels to keep the issues in this new repo separated by the various task teams.

So unless there are any objections I can take the following steps to resolve this:

1. delete all the empty repos (the ones which were never used like MetaDataTaskTeam and PCAP)
2. rename this repo to "schemas" to preserve all of its history
3. make 2 new labels "ReadsTaskTeam" and "RefVariationTaskTeam" (the active teams)
4. tag all of our issues as "ReadsTaskTeam"
5. the variants team can then move their code and issues into the schemas repo as their leisure. (there are only 5 issues and they are all pull requests so this should be straight forward)

I'll plan to take the first 4 steps this afternoon (everything listed is still reversible at this point)

Please shout if there are objections!

from ga4gh-schemas.

You might want to flatten the repository hierarchy a bit while you're at
it. If other people want to write schemas based on our schemas, they'd want
to pull this in a Git submodule inside their own Avro schemas directory,
and it would help for the path names they need to import to be as short as
possible.

-Adam

On Mon, May 12, 2014 at 11:54 AM, cassiedoll [email protected]:

It looks like everyone is in agreement here on ending up with:

• one "schemas" repo in ga4gh which will contain all the avdl files.
• we'll use labels to keep the issues in this new repo separated by
  the various task teams.

So unless there are any objections I can take the following steps to
resolve this:

• delete all the empty repos (the ones which were never used like
  MetaDataTaskTeam and PCAP)
• rename this repo to "schemas" to preserve all of its history
• make 2 new labels "ReadsTaskTeam" and "RefVariationTaskTeam" (the
  active teams)
• tag all of our issues as "ReadsTaskTeam"
• the variants team can then move their code and issues into the
  schemas repo as their leisure. (there are only 5 issues and they are all
  pull requests so this should be straight forward)

I'll plan to take the 4 steps this afternoon (everything listed is still
reversible at this point)

Please shout if there are objections!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40#issuecomment-42872589
.

from ga4gh-schemas.

Steps 1-4 don't involve moving code.
Let's get this first part done and then we can have a pull request for collapsing the hierarchy (which I'm also in favor of)

from ga4gh-schemas.

+1 to step 1-4. I agree with @adamnovak, but let's resolve 1-4 first as @cassiedoll suggested.

from ga4gh-schemas.

2-4 are done.
For 1 - I've submitted #45 so that the FileFormatsTaskTeam repo can be removed (the only commit is the readme)

The Beacon repo (3 markdown files) and the RefVariationTaskTeam repo (many files) will both need merging by someone on those teams.

Separately our readme should be improved to include all task team descriptions, I'll leave that to someone else who wants to play with words :)

from ga4gh-schemas.

I'm +1 for having all the schemas in a single repository, keeping the API separate from implementations and publishing artifacts to Maven.

Another advantage of a single repo is that we'll be able to test all the schemas together using Travis in order to make sure that schema dependencies are never broken.

One thing to keep in mind -- for now we should focus on publish Java artifacts via Maven but we'll also likely want to publish a C/C++ library as well.

from ga4gh-schemas.

Excellent.
@benedictpaten is going to propose a pull request for merging the variation work over (part of #41)

After that I think there are 2 cleanup tasks left:

• figuring out what to do with the Beacon repo
• restructuring the website to reduce confusion

And someone should probably file a pull request for:

• moving the avdl files up a couple of directories

from ga4gh-schemas.

+1 to "moving the avdl files up a couple of directories".

from ga4gh-schemas.

+1. What about making the avro/ directory somewhat hierarchical, to make it
easier to distinguish the refVar (for example) graph schemas from the read
group schemas?

On Wed, May 14, 2014 at 12:27 PM, Heng Li [email protected] wrote:

+1 to "moving the avdl files up a couple of directories".

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40#issuecomment-43126040
.

from ga4gh-schemas.

FYI: This issue now tracks the deletion of the RefVariationTaskTeam repo: https://github.com/ga4gh/RefVariationTaskTeam/issues/8

from ga4gh-schemas.

What about making the avro/ directory somewhat hierarchical, to make it easier to distinguish the refVar (for example) graph schemas from the read group schemas?

I mildly prefer to put all schema files in the same directory. This seems cleaner and easier for outsiders who don't know the differences between task teams. Nonetheless, I don't have a strong opinion. I am happy to go with whatever the consensus is.

from ga4gh-schemas.

+1 on separating the different APIs to some extent. We probably want a
couple of different .avdl files for the reference variation API (we have a
whole hierarchy system to implement that's not in there yet), and it would
be good to be able to see at a glance what .avdls belong to what API.

I would want /schemas/reads, /schemas/refvar, and so on, if
possible.
On May 14, 2014 12:29 PM, "Benedict Paten" [email protected] wrote:

+1. What about making the avro/ directory somewhat hierarchical, to make
it
easier to distinguish the refVar (for example) graph schemas from the read
group schemas?

On Wed, May 14, 2014 at 12:27 PM, Heng Li [email protected]
wrote:

+1 to "moving the avdl files up a couple of directories".

—
Reply to this email directly or view it on GitHub<
https://github.com/ga4gh/schemas/issues/40#issuecomment-43126040>
.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40#issuecomment-43126217
.

from ga4gh-schemas.

Re directory depth -- for those who weren't on today's call -- @massie made a case for keeping the current structure (/schemas/src/main/resources/avro) -- I'll let him repeat it here.

Re sub-directories by sub-APIs -- I'm +0; it feels slightly tidier, but I'd also be okay waiting until we have a dozen or so files before worrying about it. (Note that we'll also want to keep some files in the root, and/or add a /common folder.)

from ga4gh-schemas.

On Wed, May 14, 2014 at 5:08 PM, David Glazer [email protected]:

Re directory depth -- for those who weren't on today's call -- @massiehttps://github.com/massiemade a case for keeping the current structure (
/schemas/src/main/resources/avro) -- I'll let him repeat it here.

I don't think it's a big deal, providing the API website is easy to find.

Re sub-directories by sub-APIs -- I'm +0; it feels slightly tidier, but
I'd also be okay waiting until we have a dozen or so files before worrying
about it. (Note that we'll also want to keep some files in the root, and/or
add a /common folder.)

Yeah, I agree, I think we'll combine the referenceVariation schema files
into one file for now, like the reads group. We need to start simple and
make it easy for people to get the info, later we can split things up
hierarchically.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40#issuecomment-43155515
.

from ga4gh-schemas.

two comments:

1. re source avro files in /schemas/src/main/resources/avro is a maven convention, but it is easily changed in the pom.xml file if there is an overriding need to include this repository as a git submodule to other projects. My vote would be to keep the current structure until we have to cross the repo-integration bridge.
2. -1 on project subdirs. I think a single dir is good enough for our purposes

from ga4gh-schemas.

Closing -- per Wednesday's call people are comfortable with the current state of site nav, so the original goal of this issue has been accomplished. If anyone has specific further changes they'd like, let's open them up as new issues (or better yet new pull requests).

from ga4gh-schemas.