Structure of json metatdata file about brainglobe-atlasapi HOT 29 CLOSED

I think this looks good. Do we have a plan of how to deal with atlases that are defined in terms of meshes/volumes/both?

from brainglobe-atlasapi.

I don't know, would be happy to hear some inputs there. I have been working only with volumes so far. I don't know how much space reasonably resolved meshes would take (@FedeClaudi?) but if the volume's not too big, I would actually automatically create them for any volume we might have (or viceversa, if needed). I also don't know - out of inexperience - how much of that can be automatic and how much can be manually curated; I assume some level of curation is required?

from brainglobe-atlasapi.

The meshes are usually pretty small, not sure how big they are for a whole brain.

Automatic generation is pretty good. I would think there would be minimal manual curation, maybe just parameter tweaking on an atlas-by-atlas basis.

The only issue that I've found when creating a full set of meshes from an image-based atlas, is that they don't "fit" together as well as those generated directly from vector drawings (the meshes might need to be smoothed etc.). This is a minor problem, but just to say that the results may not always look as nice as the allen meshes.

from brainglobe-atlasapi.

Meshes are small. All the meshes for the allen brain atlas (which is a lot of stuff) are less than 200MB, and they can be downloaded based on the needs.

I think we should aim to have both volumetric and meshes. Volumetric is better for some analysis but you need meshes for visualization. Regarding the conversion:

• Volumetric -> meshes is fairly easy as Adam said
• Meshes -> volumetric should also be fairly doable

For atlases that are already developed (zfish, drosophila, mouse..) there should be already both kinds of data. For others I wouldn't worry too much about it not looking to good for now.

from brainglobe-atlasapi.

So, would you be happy with the following solution?

1. We keep in the directory that is downloaded (i.e., in the remote directory we will curate) both meshes and volumetric;
2. if either of those in not provided, we generate it;
3. We provide to the users functions to go from one to the other for their custom atlases.

Also @FedeClaudi , I was thinking of a single transformation matrix (i.e., resolution etc) for all the data, as I would assume that all the stacks in the folder by definition should be in the same space. This is why I did not have a resolution field for each stack. Do you think this would not be a valid assumption and that is better to have the field for all the files?

from brainglobe-atlasapi.

I agree, but I think it would be good to still have it as a metadata parameter.

from brainglobe-atlasapi.

I agree, but I think it would be good to still have it as a metadata parameter.

I would have it as a metadata entry, just at the folder level and not at the single stack level

from brainglobe-atlasapi.

So, I have been working on this a bit more. I think that Allen methods to work with regions hierarchical structure make a lot of sense and are pretty compact, so I just included the two modules (with related licence) so we can use them internally without all the AllenSDK overhead.

Now the class starts having some content with all the fundamental methods for:

• initialising the atlas;
• reading lazily the volumes,
• checking region/hemisphere from coordinates,
• loading and caching meshes from obj files

For obj files, I currently used the AllenSDK dedicated function as pymesh was having troubles with Allen's .obj but I have no great experience with such files - feel free to suggest any module for reading the meshes.

I think that it make sense to handling the remote fetching in default or custom directory in a separate subclass.

from brainglobe-atlasapi.

I added a draft for a concise way to handle downloading. With the current structure, each atlas class (one for each animal/atlas type in the future) has a name from which the the download URL from gin can be generated. Download is triggered when instantiating a BrainGlobeAtlas object, if it has not been downloaded before.

e.g.:

from brainatlas_api.bg_atlas import TestAtlas
TestAtlas()

triggers the downloading of the test atlas package in .../username/.brainglobe.

Tu use a custom location for the atlases:

from brainatlas_api.bg_atlas import TestAtlas
TestAtlas(brainglobe_path=custom_location)

What would you think about this interface and the current classes structure?

from brainglobe-atlasapi.

Hey very nice work, sorry I haven't been too active on this, might have some more time this week. The structure sounds great but I haven't checked in detail yet.

For reading .obj, vtkplotter works well for visualisations, but I guess it depends on what you need to do with them.

from brainglobe-atlasapi.

Additional point to consider @FedeClaudi @adamltyson :

If we want to (at least this is what we decided I think) have all the atlases aligned consistently, which I would find convenient, we should preserve the information of how they were rotated/flipped starting from the original stacks. There should be an entry for this in the metadata file.
Now, I was considering making it an affine matrix, with maybe some minimal class API to create it from indications like "sagittal" etc. It is much more compact , as even specifying the cuts would then also require separately the flipping, rotations, etc. etc. in different variables.
With an affine matrix, moving points back and forth from data source to brainglobe is a simple multiplication. On the other side, it won't be very readable on the metadata, but if we provide functions for the transformations maybe we don't really care?

from brainglobe-atlasapi.

that's a great idea. We can always save it as a .npy file if necessary

from brainglobe-atlasapi.

Sounds good.

Totally separate point, but we should allow for the atlases to be saved somewhere other than ~/.brainglobe, for shared machines, users without write access etc.

from brainglobe-atlasapi.

make sense. Right now that should be handled by passing a path to the atlas class no? And if none, then the package default is used. There could be a command for the user to change the package value maybe, although I never looked into how to make such meta level changes :)

from brainglobe-atlasapi.

It seems to work ok in amap/cellfinder. When you start using the software, you pass it where you would like your atlas to be saved (or where the atlas is already saved). That is then used unless you specifically update it.

If a path is isn't specified at the beginning, the default (in ~/ is used)/

from brainglobe-atlasapi.

That works for cellfinder because they're used mostly through terminal commands. In brainrender everything is done scripting by necessity so that wouldn't work, Luigi's suggestion works.

In brainrender I have default variables which can be changed like:

import brainrender
brainrender.ROOT_ALPHA = 1

and then root_alpha will have that value whenver it's used.

A more permanent solution is to have a config.json or something similar, and users can specify where to save atlases there, though it gets complicated with multiple atlases

from brainglobe-atlasapi.

It still works in a python environment, the config file is saved wherever cellfinder is installed.

from brainglobe-atlasapi.

A config.json file in the package folder is fine, as long as it is somehow exposed to the user in a decently flexible way. But still, I would expect average usage not to rely on more than 1/2 atlas per user, apart from @FedeClaudi and the flooding of supercool renderings he's looking forward to pour over the world 😃

from brainglobe-atlasapi.

A middle ground could just be to support a single "master" directory? It could be ~/.brainglobe, or it could be somewhere else. Supporting different locations could get messy.

from brainglobe-atlasapi.

ok, I would just like a way for the user to change that every moment and not just at folder installation/first usage

from brainglobe-atlasapi.

Yeah, that can be done easily with the amap/cellfinder approach. You could either:

• Pass the atlas directory to whatever you're using (if you want to use a new directory temporarily)
• Update the config file and then run with the new default

from brainglobe-atlasapi.

but if we have a single location a config file is useless, a variable such as the current brain global_api.DEFAULT_PATH is enough. I would avoid the redundant description of all atlases in a single config file as was happening in cellfinder/amap to avoid keeping multiple moving parts in synch!

from brainglobe-atlasapi.

It's a minor point, but if you want to update global_api.DEFAULT_PATH would you have to edit python code rather than a config file? This seems weird to me.

from brainglobe-atlasapi.

yum, I see the point. Let's start with a config file that contains a location path only!

from brainglobe-atlasapi.

A config.json file in the package folder is fine, as long as it is somehow exposed to the user in a decently flexible way. But still, I would expect average usage not to rely on more than 1/2 atlas per user, apart from @FedeClaudi and the flooding of supercool renderings he's looking forward to pour over the world 😃

You discovered my motivation for this project ;P

from brainglobe-atlasapi.

Should we add an optional entry for extra material/link ?
For the human atlas for instance, I would like to link the allen institute paper where they created and the paper that originally acquired the images.

from brainglobe-atlasapi.

We could either have an extra material/link entry or support multiple publications - maybe the latter is the best, as later we'll want to make sure we properly notify users of all relevant publications for the sources they're using.

So maybe just list all papers (won't generally be more than 2 I guess) separated by semicolons, and trailing square brackets specifying contribution "(atlas generation) someone et al 2015, DOI:... ; (data acquisition) someone else et al 2012, DOI:..."?

from brainglobe-atlasapi.

Should we close this? It's quite outdated anyways...

from brainglobe-atlasapi.

Definitively, we'll open new things if they arise

from brainglobe-atlasapi.