Thinking about this more and work in this repo - would we ever want a lazy flag added to the napari_get_reader hookspec?

@tlambert03 - following up on discussion in napari/napari#1023 (comment) I used cookiecutter-napari-plugin to make this plugin.

It has a folder called readers which can contain files such as imageio_reader.py, npy_reader.py, and eventually could contain mat_reader.py, pims_reader.py, meshio_reader.py, shapely_reader.py, geojson_reader.py, czi_reader.py, dv_reader.py, nd2_reader.py, bioformats_reader.py etc. all of which are very minimal wrappers / converters from underlying libraries into the format napari expects.

I think the advantage of this library is the curation and testing that we could do to ensure that files are able to be read (i'm imagining we in parallel start curating a dataset of files that could be installed and tested by our CI). Another advantage is that for napari users it becomes very simple to just install one thing and have things work, rather than having to chase down and install many things.

Note here that we're able to unify across readers for different data types, shapes, images, meshes etc. A downside is that this package could have so many many dependencies that things breaking / install problems might be common. We'd have to think about what our attitude towards submissions from others would be, we could be very liberal or conservative, especially if we though about other domains of science where file types are very different (physics, astronomy, geospatial etc.)

Alternatively we try and get napari hook impls added to the various underlying libraries directly, or we have lots of napari-io-* libraries that do the translation. Or we try and pick a smaller number of libraries to drive standardization too (i.e. get the czi reader into imageio). That still won't help with the mesh and shape reading though.

If we do do this should it live at napari/napari-io which in some senses gives it an "official branding" or should it live somewhere else?

My current opinion is to go for it at napari/napari-io, set up rigorous testing / CI and try and be as comprehensive as possible, across domains, file types, and napari layer types.

Curious what your vision for all this is @tlambert03? Also curious what @jni and @royerloic thinks.

Here is a list of some potential io packages people might want to use with napari - note this list is not exhaustive and in some places is overlapping.

For image layer:

• imageio - reads many image file formats
• PIMS - note PIMS contains many image readers inside it
• python-bioformats - note python-bioformats uses a python-javabridge to start and interact with a Java virtual machine contains many image readers
• npy, npz - simple numpy formats
• mat - load matlab files
• zarr or via xarray or via dask
• tiledb or via dask
• netCDF or via xarray
• hdf5 or via dask
• dask_image for lazy loading based on PIMS
• aicspylibczi
• nd2reader
• openslide

For surface layer:

• meshio

For shapes layer:

• geoJSON
• shapely formats - note shapely it's doesn't write or read files

For points layer:

• csv or with pandas

Following on from discussion in #1 on whether we want to maintain a library like this - a big question then is how should testing be done?

There are a variety of approaches:

• imageio has a dedicated imageio/imageio-binaries repo with small files that it uses for testing.
• meshio has a folder with small files inside it's tests folder in the main repo. I suspect those don't get put inside the actual package on pypi though.

If this repo starts depending on these libraries would we vendor these files for our own testing purposes? Sort of depends on how people feel about #1 and looking at the list of potential readers in #2 how we'd go about things.

Right now the reader_*.py files return a list of known readable extensions - maybe we should give then the full power to return a checker though - matches the generality of the napari_get_reader function.