elyx0 / s3logtools Goto Github PK

This script allows handling of S3 logs as incoming streams to be either piped and colored on the fly from the console, or to consolidate them into a file

It resorts to long polling to update the log as AWS doesn't ship in real time.

It's designed to work with logs produced by s3-streamlogger and winston.

You need to have an AWS S3 bucket and it's recommended to set a new AWS IAM user with these permissions: then store his access keys in your ~/.aws/credentials.

{
    "Version": "2012-10-17",
    "Statement": [
        
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": ["arn:aws:s3:::muzetestlogs",
                    "arn:aws:s3:::your-bucket-name/*"]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "s3:*Object",
            "Resource": ["arn:aws:s3:::your-bucket-name",
                    "arn:aws:s3:::your-bucket-name/*"]
        }
    ]
}

npm install -g s3logtools

Ensure that the AWS credentials to access your bucket are set in your environment (for example by setting the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables) or in the ~/.aws/credentials.

s3logtools --bucket=your-log-bucket-name --watch

More complex operations are possible:

s3logtools --bucket=your-log-bucket-name --ago='30 days' --output='myfile.csv' --delimiter='|'

Full list:

ISC: equivalent to 2-clause BSD.