Comments (4)
I managed to write some data. I have to check if it is correct or the order is messed up. But this is how far i have come.
auto interval1 = tensorstore::Dims(0).HalfOpenInterval(start1, end1);
auto interval2 = tensorstore::Dims(1).HalfOpenInterval(start2, end2);
auto interval3 = tensorstore::Dims(2).HalfOpenInterval(start3, end3);
std::vector<int64_t> shape = {chunkSize[0], chunkSize[1], chunkSize[2], numChannels};
auto arr = tensorstore::Array(dataPrint.data(), shape, tensorstore::fortran_order);
auto writeFuture = tensorstore::Write(tensorstore::UnownedToShared(arr), store | interval1 | interval2 | interval3);
writeFuture.commit_future.value();
auto result = writeFuture.result();
from tensorstore.
It's hard to know exactly what you want, but there are a few examples which may help you.At some point we may try and make them more prominent & expand them. See also, for example, the small benchmark we use for quick read/write test or individual tests used in the drivers:
- https://github.com/google/tensorstore/blob/master/tensorstore/examples/extract_slice.cc
- https://github.com/google/tensorstore/blob/master/tensorstore/internal/benchmark/ts_benchmark.cc
- https://github.com/google/tensorstore/blob/master/tensorstore/driver/neuroglancer_precomputed/driver_test.cc
You might try using a somewhat reduced spec to start with. neuroglancer_precomputed
should have some reasonable defaults:
Off the cuff (so, not at all tested), you want something like:
// open or create a tensorstore
TENSORSTORE_CHECK_OK_AND_ASSIGN(
auto store, tensorstore::Open({
{"driver", "neuroglancer_precomputed"},
{"kvstore",
{
{"driver", "memory"},
{"path", "prefix/"},
}},
{"multiscale_metadata",
{
{"data_type", "uint8"},
{"num_channels", 4},
{"type", "image"},
}},
{"scale_metadata",
{
{"resolution", {1, 1, 1}},
{"encoding", "raw"},
{"chunk_size", {16, 16, 16}},
{"size", {64, 64, 64}},
}},
},
tensorstore::OpenMode::open | tensorstore::OpenMode::create,
tensorstore::ReadWriteMode::read_write).result());
// shared_array is an array of uint8_t with dimensions {2,3,2,2}
auto shared_array = tensorstore::MakeArray<std::uint8_t>(
{{{{0x71, 0x72}, {0x81, 0x82}},
{{0x91, 0x99}, {0xa1, 0xa2}},
{{0xb1, 0xb2}, {0xc1, 0xc2}}},
{{{0x11, 0x12}, {0x21, 0x22}},
{{0x31, 0x32}, {0x41, 0x42}},
{{0x51, 0x52}, {0x61, 0x62}}}});
// write the array to {0,0,0,0}
TENSORSTORE_CHECK_OK(
tensorstore::Write(shared_array, store | tensorstore::AllDims().SizedInterval(
{0, 0, 0, 0}, {2, 3, 2, 2})).commit_future.result());
from tensorstore.
Thank you very much for the provided information. It was realy helpful.
Now i want to expand on this excample and use sharding. For this i found this issue: #13
It is mentioned there that work is being done on a possibility of automatically determining the parameters for the shards.
Is this already available? How can it be used?
Or should I calculate the parameters myself and integrate them into the tensorstore::open as in line 1281 of the driver_test?
from tensorstore.
Yes, there is now support for that.
If you want to specify the shard size explicitly, you can specify a write_chunk shape in the schema that is a multiple of the read chunk shape, and tensorstore will determine the corresponding sharding parameters automatically. Note, however, that the neuroglancer precomputed sharded format unfortunately has some limitations on the possible write chunk shapes due to how it is represented in terms of the morton code. For example, if your read chunk shape is (64, 64, 64, numChannels), you can make the write_chunk shape (128, 128, 128, numChannels), but not (64, 128, 64, numChannels). Since it may be tricky to figure out exactly which write chunk shapes are valid, if you don't care what it is precisely you can specify it in the schema as a soft constraint.
from tensorstore.
Related Issues (20)
- Where is the change log? HOT 2
- Registry Check fails in external package using Tensorstore as a dependency - Windows Python Wheel HOT 3
- Concatenating multiple archives HOT 8
- tensorstore cannot open vlen UTF8 string written with Zarr-Python HOT 1
- Bad Request error to access H01 dataset on a local machine HOT 2
- `zarr` driver fails to load quoted floating point data for `fill_value` HOT 1
- Can't copy or deepcopy Python TensorStore objects
- TensorStore does not compile with latest Visual Studio HOT 19
- Master does not compile on Linux HOT 11
- Tensorstore fails to compile as a CMake subproject HOT 2
- Further S3 Support Umbrella Issue HOT 3
- Converting c-order array to fortran-order array HOT 1
- Updated `bazel_to_cmake` causes trouble HOT 6
- Reading data from neuroglancer in the correct order HOT 3
- Generate `.pyi` files for type inference compatibility
- windows build failing in riegeli::EstimatedAllocatedSize HOT 16
- Replace deprecated `set-output` command with environment file HOT 1
- Any plans to implement ZEP0002 - Sharding codec? HOT 4
- Iterating over ts dataset using zarr driver does not parallelize HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tensorstore.