Look Into 1 vs Many TIFF files and Metadata about viv HOT 8 CLOSED

While it is absolutely possible to have all pyramids and metadata in the tiff file, the question is, should we? The scenario I'm thinking about is when a Tissue has hundreds (if not more) channels that were initially provided, but also new ones were added via postprocessing workflows (segmentation masks for instance) over time. We could continue to combine them into a larger and larger tiff, but for what purpose?

Alternatively, we could have one tiff per channel and a metadata file that points to all the available channels plus some useful metadata (like labels, min/max values, etc). Of course that metadata could be stored per channel, which might make sense, but I'm not sure.

So, I think its just figuring out how we'll want to do this long term and balance that with the complexity of the solution. All of it is possible. The current method of having a channel per tiff is really nice, fast, easily explainable, and can be opened in external tools easily. However we move forward, I'd like to keep those properties.

from viv.

Good points @bherr2. We were thinking about putting the channels that were acquired together into a single TIFF (that is how we are currently getting the data), but I think that post-processing data (segmentation, annotation, etc.) would be separate. The reason why I think that we should ideally keep the channels together is that it would seem preferable from a file handling point of view and would allow other TIFF viewers to operate on the same files (I presume).

from viv.

By combining them it also makes it harder to understand/process. You can definitely view the TIFFs generated by-channel now in different TIFF viewers (I tried this myself). By combining them it makes it harder to use the pyramids without knowing there are multiple channels with multiple pyramids contained within, because a standard tiff viewer may not know how to differentiate them.

We would need to somehow mark in the tiff file that 'this set of pyramids here are for channel a' and 'this set of pyramids here are for channel b'. I don't think, though I haven't looked into it very much, that there is a standard metadata element to do this and thus we'd have to add a non-standard metadata element to differentiate them. Which again, I'm not totally against it, but it would end up adding complexity to the tiff file.

Ultimately, I think we will also need external metadata so that we can point to the multiple tiffs (whether bundled by channel/source or not) and what labels to use, min/max values (potentially), etc.

from viv.

I think, having talked to some people here, the goal should be at the minimum to support both. The sample OME-TIFFs we're seeing have single-file, multi-channel, pyramids. If you have seen examples otherwise, and don't think this is the default, please let me know/post them here so I can bring them into the demo. Clearly we can support multiple files, but I think the goal should be to be compatible with the current standard, whatever it is.

from viv.

While I haven't looked at all the TIFFs on globus, I have yet to see a OME-TIFF with pyramids in it. Single-file, multi-channel, yes, but none that include image pyramids. Can you point me to one or more on globus?

from viv.

from viv.

Just as an aside, this is bigger than just the HuBMAP project. If we want to have drop-in-replacement support for pre-existing tools, this seems necessary with what we know now.

from viv.

Yeah, which is a great point. I just wasn't sure if there is a standard way to have multiple, independent pyramids and to be able to differentiate them by channel. I haven't looked closely, perhaps the OME metadata can do this?

from viv.