TransposedDataset does not support the `strides` function. about muon.jl HOT 6 CLOSED

It is not. TransposedDataset is a transposed HDF5.Dataset, the type was created to avoid reading the entire dataset into memory. So all of TransposedDataset's operations are forwarded to HDF5.Dataset, after permuting the indices if necessary. HDF5.Dataset does not define strides either.

from muon.jl.

Makes sense, except...

An H5DF Dataset is not a Julia AbstractArray. Instead it provides read and readmmap (when possible), which allows getting such an array. The result does support strides etc.

In contrast, TransposedDataset does claim to be an AbstractArray, but does not support strides. It also works similarly to an H5DF Dataset but It does not support readmmap, ismappable and iscontiguous.

So the API of TransposedDataset provides neither the complete API of an H5DF Dataset, nor the complete API for a Julia AbstractArray.

Intuitively, it should do both. That is, provide readmmap and ismappable and iscontiguous, just like an H5DF Dataset; and also provide strides (when possible), just like a Julia AbstractArray.

Naturally readmmap would need to return the Transpose of the memory-mapped matrix (using the LinearAlgebra package). That's a zero-copy view of the data so would still be memory-mapped.

from muon.jl.

strides is not part of the AbstractArray interface. AbstractArray only mandates that size and getindex are implemented, everything else is optional. strides is part of the strided Array interface.

from muon.jl.

Technically correct; however, not supporting strides for arrays which are actually strided (memory-mappable vectors and matrices) disables all sort of optimizations when actually working with these arrays.

My workaround for now is to memory-map the array using the internal dset (and transpose it), completely ignoring the fact that TransposedDataset is an AbstractArray.

Of course this runs into issue #24 so it only works for annotations written by the Python anndata package...

from muon.jl.

Note that the strided Array interface also mandates an implementation of Base.unsafe_convert, which returns a pointer to the memory block where the array is stored. This is impossible with either HDF5.Dataset or TransposedDataset.

from muon.jl.

Good point. So I guess the readmmap workaround is the only option, which depends on #24 (for data written by the Julia package).

Thanks!

from muon.jl.