Comments (6)
laramiel
commented on September 24, 2024
tensorstore is using aws v4 authentication, so it would be nice to get this to work properly.
This is not, I expect, running on EC2, but rather a local machine, correct? If you have authenticated via the aws cli, then the auth credentials should be available.
We should be reading the following environment variables
"AWS_ACCESS_KEY_ID";
"AWS_SECRET_ACCESS_KEY";
"AWS_SESSION_TOKEN";
Or looking in the file given by the environment variables:
"AWS_SHARED_CREDENTIALS_FILE", defaulting to $HOME/.aws/credentials
"AWS_PROFILE", defaulting to "default"
You could try turning on verbose logging to see if that helps:
export TENSORSTORE_VERBOSE_LOGGING=s3
from tensorstore.
sdorkenw
commented on September 24, 2024
Correct, I am running this on a local machine. I checked the environment variables and I also have a .aws/credentials file with the correct entries.
With the extra logging, I am getting more prints but not sure what to make of them yet (replaced my bucket name):
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1707173277.506206 908815 s3_resource.cc:121] Using default AdmissionQueue with limit 32
I0000 00:00:1707173277.636497 908897 s3_key_value_store.cc:1327] S3 driver using endpoint [S3EndpointHostRegion{endpoint=https://<MYBUCKET>-us-west-2.s3.us-west-2.amazonaws.com/, host_header=<MYBUCKET>-us-west-2.s3.us-west-2.amazonaws.com, aws_region=us-west-2}]
W0000 00:00:1707173277.839775 908897 curl_transport.cc:444] Error [28]=Timeout was reached in curl operation
Failed to connect to 169.254.169.254 port 80 after 201 ms: Timeout was reached
I0000 00:00:1707173277.839915 908902 s3_key_value_store.cc:496] ReadTask: HttpRequest{GET https://<MYBUCKET>-us-west-2.s3.us-west-2.amazonaws.com/test_0/zarr.json user_agent=, headers=<host: <MYBUCKET>-us-west-2.s3.us-west-2.amazonaws.com, x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, x-amz-date: 20240205T224757Z>}
I0000 00:00:1707173277.926035 908902 s3_key_value_store.cc:699] WriteTask (Peek): HttpRequest{HEAD https://<MYBUCKET>-us-west-2.s3.us-west-2.amazonaws.com/test_0/zarr.json user_agent=, headers=<if-match: "", host: <MYBUCKET>-us-west-2.s3.us-west-2.amazonaws.com, x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, x-amz-date: 20240205T224757Z>}
I0000 00:00:1707173277.944518 908897 s3_key_value_store.cc:760] WriteTask: HttpRequest{PUT https:/<MYBUCKET>-us-west-2.s3.us-west-2.amazonaws.com/test_0/zarr.json user_agent=, headers=<Content-Type: application/octet-stream, Content-Length: 433, host: <MYBUCKET>-us-west-2.s3.us-west-2.amazonaws.com, x-amz-content-sha256: aba7a6d2e78de85f18e974219f2db1afddaf3b26c920df59681585ff222b26b2, x-amz-date: 20240205T224757Z>} size=433
from tensorstore.
laramiel
commented on September 24, 2024
Ok, that looks like an anonymous S3 request, so tensorstore didn't find the credentials file or environment variables.
from tensorstore.
laramiel
commented on September 24, 2024
I've added logging. You can also try setting a credentials in the spec:
https://google.github.io/tensorstore/kvstore/s3/index.html#
"aws_credentials" : {
"profile": "default",
"filename": "/home/sdorkenw/.aws/credentials"
},
from tensorstore.
laramiel
commented on September 24, 2024
I think that this patch fragment should fix your problem.
diff --git a/tensorstore/kvstore/s3/s3_key_value_store.cc b/tensorstore/kvstore/s3/s3_key_value_store.cc
index 3d74da8..9c0c88d 100644
--- a/tensorstore/kvstore/s3/s3_key_value_store.cc
+++ b/tensorstore/kvstore/s3/s3_key_value_store.cc
@@ -235,7 +235,7 @@
const Spec& spec,
internal::ContextResourceCreationContext context) const {
auto result = GetAwsCredentialProvider(
- spec.profile, spec.filename, spec.metadata_endpoint,
+ spec.filename, spec.profile, spec.metadata_endpoint,
internal_http::GetDefaultHttpTransport());
if (!result.ok() && absl::IsNotFound(result.status())) {
return Resource{spec, nullptr};
from tensorstore.
laramiel
commented on September 24, 2024
Closing this as it now works, I think.
from tensorstore.
Related Issues (20)
- Please create a pre-built PyPI wheel for linux arm64 HOT 7
- Support of large files for grpc_kvstore HOT 1
- S3 kvstore driver not being recognized HOT 3
- downsample driver produces repeating patterns HOT 8
- Debug logs for S3 driver, log requests and responses HOT 9
- Cache pool context from multiple python processes HOT 2
- Error while installing "tensorstore" on jetson linux (jetpack) HOT 1
- Assertion `IsUnmodified()' failed HOT 6
- How to use a custom bazel installation? HOT 2
- Building on linux with ppc64le arch
- Specify S3 credentials directly HOT 3
- Transactional/ACID semantics HOT 1
- Failing to build tensorestore as a cmake project HOT 4
- Using s3 kvstore with minio HOT 3
- Error reading shard index, Requested byte range... was not satisfied by response with byte range ... HOT 4
- Unable to include tensorstore as a cmake dependency HOT 1
- Question: does tensorstore support array with multiple dynamic dimensions? HOT 2
- Clarify in documentation if the C++ API is thread safe HOT 2
- Writing local files fails on Windows 11 HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tensorstore.