s3-exporter
Prometheus metrics exporter for S3 Storage
Config example
access_key: "optional-your-access-key"
secret_key: "optional-your-secret-key"
bucket: "bucket-name"
patterns:
- "*.zip"
folders:
- "backup"
- "nextcloudbackup"
You can omit access_key and secret_key to use credentials from
environment settings, see the Boto3 documentation for more info.
If you would like metrics on objects in the root of the bucket use an empty folder, i.e.:
folders:
- ""
If you would like to cut results (i.e. w/o subfolders) use delimiter within pattern, see AWS docs for explanations, for example:
patterns:
- "some/long/path"
delimiter: "/"
Metrics
Metrics will be available at http://localhost:9327
# HELP s3_latest_file_timestamp Last modified timestamp(milliseconds) for latest file in folder
# TYPE s3_latest_file_timestamp gauge
s3_latest_file_timestamp{bucket="bucket-name",folder="backup"} 1519524066157.0
# HELP s3_oldest_file_timestamp Last modified timestamp(milliseconds) for oldest file in folder
# TYPE s3_oldest_file_timestamp gauge
s3_oldest_file_timestamp{bucket="bucket-name",folder="backup"} 1519005663854.0
# HELP s3_latest_file_size Size in bytes for latest file in folder
# TYPE s3_latest_file_size gauge
s3_latest_file_size{bucket="bucket-name",folder="backup"} 290355072.0
# HELP s3_oldest_file_size Size in bytes for latest file in folder
# TYPE s3_oldest_file_size gauge
s3_oldest_file_size{bucket="bucket-name",folder="backup"} 281699347.0
# HELP s3_file_count Numbeer of existing files in folder
# TYPE s3_file_count gauge
s3_file_count{bucket="bucket-name", folder="backup"} 7.0
# HELP s3_success Displays whether or not the listing of S3 was a success
# TYPE s3_success gauge
s3_success{bucket="bucket-name",folder="backup"} 1.0Alert Example
- Alert for time of latest backup. This example checks if backup is created every day (< 30h)
groups:
- name: host.rules
rules:
- alert: backup_is_too_old
expr: (time()) - s3_latest_file_timestamp > 108000
for: 5m
labels:
severity: critical
annotations:
description: Backup too old. Reported by instance {{ $labels.instance }}.
summary: Backup too old
- Alert for size of latest backup. This example checks latest backup file created has minimum size of 1MB
- alert: backup_size_is_too_small
expr: s3_latest_file_size < 1000000
for: 5m
labels:
severity: critical
annotations:
description: Backup size too small. Reported by instance {{ $labels.instance }}.
summary: Backup size too small
Run
Using code (local)
pip install -r app/requirements.txt
aws-profile -p prod/vault python app/exporter.py config/config.ymlUsing docker
docker run -p 9327:9327 -v ./config:/config jamotion/s3-exporter
Using docker-compose
Currently the code will not be reloaded when it changes, so run the container interactively:
docker-compose run --service-ports s3
Then run the Python exporter process:
python exporter.py /config/config.yml
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent