molu8bits / s3bucket_exporter Goto Github PK
View Code? Open in Web Editor NEWS3 bucket exporter exposing metrics size and number of objects
License: MIT License
S3 bucket exporter exposing metrics size and number of objects
License: MIT License
Counts only max 1000 objects per bucket
While i have filled the docker-compuose.yml with wasabi endpoint, etc.
When i try to run it, i am given the following error:
ysername@servername:~/s3bucket_exporter-master$ sudo docker-compose up
ERROR: The Compose file './docker-compose.yaml' is invalid because:
services.s3bucketexporter.environment.S3_DISABLE_SSL contains false, which is an invalid type, it should be a string, number, or a null
Hello @molu8bits . you said in the readme that this project was initially made to work with Ceph. im currently trying to use it for it however i kept getting empty information from my setup:
level=debug msg="s3metrics read from s3_endpoint :{Test-bucket2 true 0 0 []}"
im wondering if you know what type of capabilities the user needs to have in order to get the right information about the buckets.
also, i dont know what should be the default values for this parameters to work with ceph:
S3_DISABLE_ENDPOINT_HOST_PREFIX
S3_FORCE_PATH_STYLE
kind regards
What if you want to monitor multiple buckets spread over multiple accounts. Is there a way to do this without having to depoly multiple Dockers?
If I knew how I would submit a PR for the README because this is really hard to figure out for someone who doesn't program and just wants to monitor s3 buckets.
I've tried all sorts of combinations I found in various docs like this one and this one.
I get different errors depending on what url I try but here's on example using S3_ENDPOINT=https://bucket-name.eu-central-1.s3.amazonaws.com:443
;
time="2021-05-11T21:34:15Z" level=info msg="Beginning to serve on port:9655"
time="2021-05-11T21:34:15Z" level=info msg="s3 name 'eu-central-1' available at s3 endpoint 'https://bucket-name.eu-central-1.s3.amazonaws.com:443' will be monitored"
time="2021-05-11T21:34:15Z" level=info msg="listenPort ::9655"
Connection to S3 endpoint failed : RequestError: send request failed
caused by: Get https://bucket-name.eu-central-1.s3.amazonaws.com:443/: x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not bucket-name.eu-central-1.s3.amazonaws.com
panic: inconsistent label cardinality: expected 1 label values but got 0 in []string(nil)
goroutine 10 [running]:
github.com/prometheus/client_golang/prometheus.MustNewConstMetric(...)
/go/src/github.com/prometheus/client_golang/prometheus/value.go:107
main.S3Collector.Collect(0xc000088540)
/build/main.go:84 +0xd3b
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
/go/src/github.com/prometheus/client_golang/prometheus/registry.go:446 +0x19d
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
/go/src/github.com/prometheus/client_golang/prometheus/registry.go:457 +0x571
Hi,
I don't manage to make it work on a s3 compatible storage that is not AWS.
On my local computer on windows 10, docker desktop and Git Bash, I do :
docker run -p 9655:9655 -d -e LISTEN_PORT=:9655 -e S3_DISABLE_SSL=False -e S3_ENDPOINT=https://s3.waw2-1.cloudferro.com -e S3_ACCESS_KEY=xxxxxxxxx -e S3_SECRET_KEY=xxxxxxxxxxxxxxxx -e S3_NAME=s3_cloudferro -e S3_DISABLE_ENDPOINT_HOST_PREFIX=True -e LOG_LEVEL=Debug -e S3_FORCE_PATH_STYLE=True docker.io/molu8bits/s3bucket_exporter:1.0.2
Then I do:
curl -v http://127.0.0.1:9655/metrics
I get:
* Trying 127.0.0.1:9655...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 127.0.0.1 (127.0.0.1) port 9655 (#0)
> GET /metrics HTTP/1.1
> Host: 127.0.0.1:9655
> User-Agent: curl/7.84.0
> Accept: */*
>
0 0 0 0 0 0 0 0 --:--:-- 0:07:56 --:--:-- 0
In the logs I have:
time="2024-04-12T10:13:08Z" level=info msg="Beginning to serve on port:9655"
time="2024-04-12T10:13:08Z" level=info msg="s3 name 's3_wekeo' available at s3 endpoint 'https://s3.waw2-1.cloudferro.com' will be monitored"
time="2024-04-12T10:13:08Z" level=info msg="listenPort ::9655"
In the docker statistics, after one minute or so, the CPU usage increases and the network input increases by ~3 Mb/sec, reaching 500 Mb after like 2-3 minutes.
S3_NAME is not supposed to match with something, right ? It's just a string that I choose.
Do you see any mistake ?
Thank you
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.