Memory mapping along with compression about hdf5.jl HOT 6 CLOSED

mmap isn't magic; it just makes a file act like a block of memory, the same way swap/virtual memory do. It requires that the in-memory representation is identical to the on-disk representation. This is a limitation at the OS level.

You might be able to use the hyperslab interface, which doesn't use mmap, to read subsets of a compressed array, although I haven't tried it.

from hdf5.jl.

Hi Simon,

I see, the package is using mmap directly.

Pytables (http://pytables.github.io/usersguide/optimization.html#compression-issues) has interesting facilities for compressing data, some that are not on the HDF5 standard, and that work with lazy loading. But it does not use mmap directly, from what I understand it has its own implementation (http://comments.gmane.org/gmane.comp.python.pytables.user/2903).

Is the hyperslab interface implemented in HDF5.jl?

Thanks,
Cássio

from hdf5.jl.

Yes. See the docs.

from hdf5.jl.

Can you point me to it? https://github.com/timholy/HDF5.jl/blob/master/doc/hdf5.md does not mention hyperslab.

I'd like to read an array written as f["A", "chunk", (10000,1), "compress", 3] = A using lazy loading.

Thanks,
Cássio

from hdf5.jl.

That's what this section describes (the Asub = dset[2:3, 1:3] syntax).

from hdf5.jl.

Thanks Tim. I thought there was something else to it.

from hdf5.jl.