Comments (1)
I will want to spend some time thinking about this.
There are two objections that immediately come to mind:
- most operations within kerchunk work on the content of keys, so they will always be working at the dict level to directly set values. The mapper and zarr views necessarily prevent this.
- during combine, we now support writing directly to parquet. The interface is still store-like, but the access pattern is very different; so it's not a case of "build the dicts, then serialise to parquet", but "serialise to parquet on the fly" (in order to save memory).
So maybe it could be the other way around: the reference sets, dict-like stores, acquire .to_zarr and .to_mapper methods which use the information already contained within, but the primary representation is still dicts.
from kerchunk.
Related Issues (20)
- kerchunking zarr from OSN, bucket not found HOT 2
- Parquet reference files from git with simplecache HOT 1
- Release? HOT 4
- Kerchunk doesn't translate HDF5 hard links HOT 3
- add to xarray backends docs HOT 24
- grib_tree's unexpected behaviour HOT 5
- Support the Open Meteo custom data format HOT 9
- NetCDF file has one time step, kerchunk-generated reference has nine time steps? HOT 3
- How to use subchunking HOT 10
- inline_threshold not encoding time value? HOT 9
- Got PermissionError: Access Denied when trying the example from the tutorial HOT 8
- KeyError when using Parquet-converted JSON reference for NetCDF4 access over HTTP HOT 12
- Chunk size mismatch on concatenation HOT 3
- MultiZarrToZarr append method - coo_map not working as expected HOT 5
- No module named 'eccodes' when trying to read kerchunked grib dataset with remote Dask cluster HOT 2
- Error with combining kerchunk mappings with MultiZarrToZarr HOT 2
- Unable to append to already existing Kerchunk parquet store HOT 6
- Compressor is stored under filters in kerchunk >= 0.2.1 HOT 4
- Nested HDF5 Data / HEC-RAS HOT 4
- Inconsistent behavior between Parquet and JSON when chunks are missing HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kerchunk.