Comments (9)
martindurant
commented on July 18, 2024
Please ping me again after pydata madness has calmed down a bit.
It would be worthwhile debuging to see what value is in the line that failed (perhaps None?).
from kerchunk.
zcobell
commented on July 18, 2024
I've updated the above notebook to show what seems like it could be a bug in refs_to_dataframe with a trivial fix, however, I'll reserve stating that definitively until I am able to solve the original error in this topic. The quick summary is:
out = LazyReferenceMapper.create(
record_size, root=url, fs=fs, categorical_threshold=categorical_threshold
)should become:
out = LazyReferenceMapper.create(
record_size=record_size, root=url, fs=fs, categorical_threshold=categorical_threshold
)to avoid:
TypeError: LazyReferenceMapper.create() got multiple values for argument 'root'from kerchunk.
zcobell
commented on July 18, 2024
@martindurant - certainly. Thanks!
from kerchunk.
martindurant
commented on July 18, 2024
Yes, I'm sure the signature and docs of that method can be tightened up. In any case, explicit is good.
from kerchunk.
zcobell
commented on July 18, 2024
Adding this while it's fresh and then I'll resurface this toward the end of the week.
The value at that point is a list pointing to the .zarray file, but it appears that the __setitem__ method expects it to have been read into a dict already.
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr.json
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/.zgroup
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/.zmetadata
2023-12-06T11:05:31EST :: DEBUG :: selector_events.py :: __init__ :: Using selector: KqueueSelector
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_3.zarr.json
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/.zgroup
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/.zmetadata
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/parquet/.zmetadata
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/parquet/.zmetadata
2023-12-06T11:05:31EST :: DEBUG :: combine.py :: __init__ :: Concat dims: ['storm']
2023-12-06T11:05:31EST :: DEBUG :: combine.py :: __init__ :: Coord map: {'storm': [1, 2]}
2023-12-06T11:05:31EST :: DEBUG :: combine.py :: fss :: setup filesystems
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr.json
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_3.zarr.json
2023-12-06T11:05:31EST :: DEBUG :: combine.py :: first_pass :: First pass: 0
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: .zgroup
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: .zgroup, type bytes
2023-12-06T11:05:31EST :: DEBUG :: combine.py :: _get_value :: Decode: ([1, 2], 0, 'storm', 'file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr.json') -> 1
2023-12-06T11:05:31EST :: DEBUG :: combine.py :: first_pass :: First pass: 1
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: .zgroup
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: .zgroup, type bytes
2023-12-06T11:05:31EST :: DEBUG :: combine.py :: _get_value :: Decode: ([1, 2], 1, 'storm', 'file:///Users/zcobell/Documents/Code/model2zarr/result_3.zarr.json') -> 2
2023-12-06T11:05:31EST :: DEBUG :: combine.py :: first_pass :: Created coordinates map
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: __setitem__ :: Setting .zgroup to b'{\n "zarr_format": 2\n}'
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: .zgroup
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: .zgroup, type bytes
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: __setitem__ :: Setting storm/.zarray to b'{\n "chunks": [\n 2\n ],\n "compressor": null,\n "dtype": "<i8",\n "fill_value": null,\n "filters": null,\n "order": "C",\n "shape": [\n 2\n ],\n "zarr_format": 2\n}'
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: __setitem__ :: Setting storm/.zattrs to b'{\n "_ARRAY_DIMENSIONS": [\n "storm"\n ]\n}'
2023-12-06T11:05:31EST :: DEBUG :: combine.py :: store_coords :: Written coordinates
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: .zgroup
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: .zgroup, type bytes
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: __setitem__ :: Setting .zgroup to {"zarr_format":2}
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: .zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: .zattrs, type bytes
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: __setitem__ :: Setting .zattrs to {"coordinates":"latitude longitude"}
2023-12-06T11:05:31EST :: DEBUG :: combine.py :: store_coords :: Written global metadata
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: .zgroup
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: .zgroup, type bytes
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: air_pressure_at_sea_level/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: air_pressure_at_sea_level/.zarray, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/air_pressure_at_sea_level/.zarray']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/air_pressure_at_sea_level/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: air_pressure_at_sea_level/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: air_pressure_at_sea_level/.zattrs, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/air_pressure_at_sea_level/.zattrs']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/air_pressure_at_sea_level/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: eastward_water_velocity/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: eastward_water_velocity/.zarray, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/eastward_water_velocity/.zarray']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/eastward_water_velocity/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: eastward_water_velocity/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: eastward_water_velocity/.zattrs, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/eastward_water_velocity/.zattrs']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/eastward_water_velocity/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: eastward_wind/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: eastward_wind/.zarray, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/eastward_wind/.zarray']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/eastward_wind/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: eastward_wind/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: eastward_wind/.zattrs, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/eastward_wind/.zattrs']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/eastward_wind/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: element/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: element/.zarray, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/element/.zarray']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/element/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: element/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: element/.zattrs, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/element/.zattrs']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/element/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: latitude/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: latitude/.zarray, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/latitude/.zarray']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/latitude/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: latitude/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: latitude/.zattrs, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/latitude/.zattrs']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/latitude/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: longitude/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: longitude/.zarray, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/longitude/.zarray']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/longitude/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: longitude/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: longitude/.zattrs, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/longitude/.zattrs']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/longitude/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: northward_water_velocity/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: northward_water_velocity/.zarray, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/northward_water_velocity/.zarray']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/northward_water_velocity/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: northward_water_velocity/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: northward_water_velocity/.zattrs, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/northward_water_velocity/.zattrs']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/northward_water_velocity/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: northward_wind/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: northward_wind/.zarray, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/northward_wind/.zarray']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/northward_wind/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: northward_wind/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: northward_wind/.zattrs, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/northward_wind/.zattrs']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/northward_wind/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: sea_surface_height_above_geoid/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: sea_surface_height_above_geoid/.zarray, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/sea_surface_height_above_geoid/.zarray']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/sea_surface_height_above_geoid/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: sea_surface_height_above_geoid/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: sea_surface_height_above_geoid/.zattrs, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/sea_surface_height_above_geoid/.zattrs']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/sea_surface_height_above_geoid/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: time/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: time/.zarray, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/time/.zarray']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/time/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: time/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: time/.zattrs, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/time/.zattrs']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/time/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: combine.py :: _get_value :: Decode: ([1, 2], 0, 'storm', 'file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr.json') -> 1
2023-12-06T11:05:31EST :: DEBUG :: combine.py :: second_pass :: Second pass: 0, air_pressure_at_sea_level
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: air_pressure_at_sea_level/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: air_pressure_at_sea_level/.zarray, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/air_pressure_at_sea_level/.zarray']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/air_pressure_at_sea_level/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: air_pressure_at_sea_level/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: air_pressure_at_sea_level/.zattrs, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/air_pressure_at_sea_level/.zattrs']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/air_pressure_at_sea_level/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: __setitem__ :: Setting air_pressure_at_sea_level/.zarray to {"chunks":[1,252,130],"compressor":{"blocksize":0,"clevel":5,"cname":"lz4","id":"blosc","shuffle":1},"dtype":"<f8","fill_value":null,"filters":null,"order":"C","shape":[2,252,8303],"zarr_format":2}
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: __setitem__ :: Setting air_pressure_at_sea_level/.zattrs to {"_ARRAY_DIMENSIONS":["storm","time","node"],"coordinates":"time y x","location":"node","long_name":"air pressure at sea level","mesh":"adcirc_mesh","standard_name":"air_pressure_at_sea_level","units":"meters of water"}
2023-12-06T11:05:31EST :: DEBUG :: combine.py :: second_pass :: Second pass: 0, eastward_water_velocity
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: eastward_water_velocity/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: eastward_water_velocity/.zarray, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/eastward_water_velocity/.zarray']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/eastward_water_velocity/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: eastward_water_velocity/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: eastward_water_velocity/.zattrs, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/eastward_water_velocity/.zattrs']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/eastward_water_velocity/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: __setitem__ :: Setting eastward_water_velocity/.zarray to {"chunks":[1,252,130],"compressor":{"blocksize":0,"clevel":5,"cname":"lz4","id":"blosc","shuffle":1},"dtype":"<f8","fill_value":null,"filters":null,"order":"C","shape":[2,252,8303],"zarr_format":2}
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: __setitem__ :: Setting eastward_water_velocity/.zattrs to {"_ARRAY_DIMENSIONS":["storm","time","node"],"coordinates":"time y x","dry_Value":-99999.0,"location":"node","long_name":"water column vertically averaged east\/west velocity","mesh":"adcirc_mesh","positive":"east","standard_name":"eastward_water_velocity","units":"m s-1"}
2023-12-06T11:05:31EST :: DEBUG :: combine.py :: second_pass :: Second pass: 0, eastward_wind
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: eastward_wind/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: eastward_wind/.zarray, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/eastward_wind/.zarray']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/eastward_wind/.zarray
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: cat: eastward_wind/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: _cat_common :: Reference: eastward_wind/.zattrs, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/eastward_wind/.zattrs']
2023-12-06T11:05:31EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/eastward_wind/.zattrs
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: __setitem__ :: Setting eastward_wind/.zarray to {"chunks":[1,252,130],"compressor":{"blocksize":0,"clevel":5,"cname":"lz4","id":"blosc","shuffle":1},"dtype":"<f8","fill_value":null,"filters":null,"order":"C","shape":[2,252,8303],"zarr_format":2}
2023-12-06T11:05:31EST :: DEBUG :: reference.py :: __setitem__ :: Setting eastward_wind/.zattrs to {"_ARRAY_DIMENSIONS":["storm","time","node"],"coordinates":"time y x","location":"node","long_name":"e\/w wind velocity","mesh":"adcirc_mesh","positive":"east","standard_name":"eastward_wind","units":"m s-1"}
2023-12-06T11:05:32EST :: DEBUG :: combine.py :: second_pass :: Second pass: 0, element
2023-12-06T11:05:32EST :: DEBUG :: reference.py :: _cat_common :: cat: element/.zarray
2023-12-06T11:05:32EST :: DEBUG :: reference.py :: _cat_common :: Reference: element/.zarray, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/element/.zarray']
2023-12-06T11:05:32EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/element/.zarray
2023-12-06T11:05:32EST :: DEBUG :: reference.py :: _cat_common :: cat: element/.zattrs
2023-12-06T11:05:32EST :: DEBUG :: reference.py :: _cat_common :: Reference: element/.zattrs, whole file => ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/element/.zattrs']
2023-12-06T11:05:32EST :: DEBUG :: local.py :: __init__ :: open file: /Users/zcobell/Documents/Code/model2zarr/result_2.zarr/element/.zattrs
2023-12-06T11:05:32EST :: DEBUG :: reference.py :: __setitem__ :: Setting element/.zarray to {"chunks":[1,14761,3],"compressor":{"blocksize":0,"clevel":5,"cname":"lz4","id":"blosc","shuffle":1},"dtype":"<i4","fill_value":null,"filters":null,"order":"C","shape":[2,14761,3],"zarr_format":2}
2023-12-06T11:05:32EST :: DEBUG :: reference.py :: __setitem__ :: Setting element/.zattrs to {"_ARRAY_DIMENSIONS":["storm","nele","nvertex"],"cf_role":"face_node_connectivity","long_name":"element","start_index":1,"units":"nondimensional"}
2023-12-06T11:05:32EST :: DEBUG :: reference.py :: __setitem__ :: Setting latitude/.zarray to ['file:///Users/zcobell/Documents/Code/model2zarr/result_2.zarr/latitude/.zarray']
Traceback (most recent call last):
File "/Users/zcobell/Documents/Code/model2zarr/./chunk.py", line 40, in <module>
ref = mzz.translate()
^^^^^^^^^^^^^^^
File "/Users/zcobell/miniconda3/envs/model2zarr/lib/python3.12/site-packages/kerchunk/combine.py", line 500, in translate
self.second_pass()
File "/Users/zcobell/miniconda3/envs/model2zarr/lib/python3.12/site-packages/kerchunk/combine.py", line 395, in second_pass
self.out[k] = fs.references[k]
~~~~~~~~^^^
File "/Users/zcobell/miniconda3/envs/model2zarr/lib/python3.12/site-packages/fsspec/implementations/reference.py", line 363, in __setitem__
self.zmetadata[key] = json.loads(
^^^^^^^^^^^
TypeError: Expected string or C-contiguous bytes-like object
from kerchunk.
martindurant
commented on July 18, 2024
So it seems we assume that the .zarray is always inlined, which would be true for any kerchunked dataset where we create the file (from hdf, grib, etc) with the sole exception of existing zarr datasets where they already exist. We could cover this case, but I would argue that ZarrToZarr should always inline .z* files as another way to solve it.
from kerchunk.
zcobell
commented on July 18, 2024
Got it. I might be able to take a crack at implementing this over the next day or so if for no other reason than to bump into some walls to familiarize myself with some of the internals. If there was a particular way you'd had in mind to do this, I'm all ears.
from kerchunk.
zcobell
commented on July 18, 2024
Updating the inlining to use the following code in do_inline in utils.py works well, but does slightly feel hacky. I'm guessing there may be a more robust way to do it than this. This has the effect of finding the *.z files that come in as a list object and always inlining them
get_keys = [
k
for k, v in out.items()
if (isinstance(v, list) and len(v) == 3 and v[2] < threshold)
or (
isinstance(v, list)
and len(v) == 1
and isinstance(v[0], str)
and v[0].split("/")[-1].startswith(".z")
)
]from kerchunk.
martindurant
commented on July 18, 2024
I am quite sure that every scanner except zarr already inlines all .z files (or creates them) and has special code for this.
I think the code is fine; the two conditions could be annotated with comments ("small enough", "metadata").
from kerchunk.
Related Issues (20)
- Coordinate Variables have incorrect shapes and chunks if name matches a dimension
- release? HOT 6
- Status of JSON Schema for references spec HOT 1
- Dataclass for "VirtualZarrStore" HOT 1
- Refactor file format backend openers HOT 1
- Refactor MultiZarrToZarr into multiple functions HOT 24
- Memory leak in `scan_grib`? HOT 2
- v0.2.1 broke existing workflows - 'SingleHdf5ToZarr' object has no attribute 'decoders' HOT 2
- Issue reading GZIP compressed GRIB2 remote file HOT 6
- _FillValue from a classic netCDF file does not end up in the kerchunk json HOT 4
- [BUG] MultiZarrToZarr fails to concatenate dimension coordinate variables HOT 10
- SingleHdf5ToZarr error/warning with scalar utf-8/ascii dataset HOT 4
- Failure to combine multiple JSON reference files via MultiZarrToZarr() HOT 23
- Best practice for using `rename_target_files` HOT 5
- Data for variable is not inlined despite size below threshold HOT 8
- Tests failing after upstream change to LazyReferenceMapper
- Unexpected behavior with MultiZarrToZarr with partial chunks. HOT 1
- Grib2 Split File with read ahead? HOT 2
- multiple reads of nasa-nex-gddp-cmip6 dataset from MultiZarrToZarr concatenated metadata returns all nans HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kerchunk.