Improve Chest about dask HOT 6 CLOSED

A question: how constrained can Chest's API be in terms of types?

I ask because I think it would be interesting/fun/useful to implement either Thrift or Protobuf (or maybe we could try both and benchmark them) as a backend for Chest and be able to avoid pickle, because it's pickle.

Another thought: If we only need mapping between keys and arrays, we might be able to create a custom serialization format around bcolz, and then we get all the benefits that entails.

Thoughts? This is a problem I'm interested in hacking on.

from dask.

One could parametrize chest with strict key and value types and do some type checking. Not sure if this should live outside or inside core Chest class.

Chest currently can be parametrized by dump/load functions so swapping in other protocols can be made to work that way.

One reason that I actually like pickle is that it handles numpy and pandas very nicely (as long as you use pickle.dump(..., protocol=2)). I'm pretty confident that this does almost no manipulation of the actual data bytes (though I'm sure that the metadata is entirely mangled).

Using BColz (or compression generally) also makes sense. One thing to beware of is that bcolz often trades CPU for Memory bandwidth. In the context of dask we may not have excess CPU to spare and minimally compressed may be better than in a normal, single-core case.

from dask.

@wrobstory if you're at all interested in any of this I'd be thrilled.

I think that efficient spill-to-disk data structures is a potentially very useful and relatively untapped development space. I'd love to see someone revamp chest or even reinvent something newer and better (chest was a quick-and-dirty evening project).

from dask.

I've added a naive LRU solution for chest in mrocklin/chest@24210a5

from dask.

And significantly reduced write costs by being less dumb in mrocklin/chest@ba15e5e

from dask.

And it's now threadsafe (maybe). I'm going to close this for now. Chest meets immediate requirements for use with numpy+dask things. Blogpost forthcoming.

from dask.