duo-labs / cloudtrail-partitioner Goto Github PK
View Code? Open in Web Editor NEWLicense: BSD 3-Clause "New" or "Revised" License
License: BSD 3-Clause "New" or "Revised" License
My Partitioner function is failing with the error below:
[ERROR] Runtime.ImportModuleError: Unable to import module 'main': No module named 'yaml'
I commented out the import yaml line, as well as the lines that try to read the yaml config file, and it seems to have resolved the issue.
It looks like a week or so ago AWS released partition projection for Athena. It discusses the performance benefits (apparently it doesn't need to call the Glue API to list partitions) but I feel the real benefit is avoiding the need to create partitions.
For example, here's how I set it up for my CloudTrail logs:
CREATE EXTERNAL TABLE cloudtrail_logs_auto (
eventversion STRING,
-- ...trimmed for clarity
sharedeventid STRING,
vpcendpointid STRING
)
PARTITIONED BY (accountId string, region string, date string)
ROW FORMAT SERDE 'com.amazon.emr.hive.serde.CloudTrailSerde'
STORED AS INPUTFORMAT 'com.amazon.emr.cloudtrail.CloudTrailInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION 's3://bucketname/AWSLogs/o-orgid/'
TBLPROPERTIES(
"projection.enabled" = "true",
"projection.date.type" = "date",
"projection.date.range" = "2020/01/01,NOW",
"projection.date.format" = "yyyy/MM/dd",
"projection.date.interval" = "1" ,
"projection.accountid.type" = "enum",
"projection.accountid.values" = "0123456789012,210987654321,etc",
"projection.region.type" = "enum",
"projection.region.values" = "us-east-1,us-west-2,ap-southeast-2,etc",
"storage.location.template" = "s3://bucketname/AWSLogs/o-orgid/${accountid}/CloudTrail/${region}/${date}"
);
This allows me to run queries like SELECT * FROM cloudtrail_logs_auto where date >= '2020/02/28' and date < '2020/06/01'
, with or without accountId
and region
columns, etc. A few things worth noting:
NOW
in projection.date.range
means "now" in terms of when the query is executed. So new projected partitions automatically appear each day.SHOW partitions cloudtrail_logs_auto
will yield no resultsI haven't yet found a gotcha or a reason why not to use this new functionality. As long as there aren't any show-stopper issues, it feels like this changes the need for cloudtrail-partitioner
: it could instead change to keeping the projection.accountid.values
and projection.region.values
properties up to date instead. Thoughts?
AWS recently launched a service which wraps a lot of this up for you: CloudTrail Lake
Does it make sense to point people there up front and then they can leverage this project if they want to manage things themselves?
Ensure the query results bucket exists, and is in this account (to detect bucket sniping attacks)
This would be a big change for this project. Athena falls over when it tries to read too many small files (it crashes due to rate limiting apparently). CloudTrail log files are often a few KB in size in less active accounts. Athena works best when it reads files that are 64MB apparently. A nightly ETL job could take the previous day's log files and concat them into 64MB files, possibly into a separate S3 bucket.
I'm unsure of doing this. This was part of feedback I received from the Athena team for problems I was running into with a client. I'm more in the camp that Athena should be fixed, and not that I need to build an ETL to work around its limitations.
In an Organization trail, if you have account 000000000000 as your Org master, and say 111111111111 as another account, then your S3 bucket will contain:
- AWSLogs
- 000000000000
- o-123
- 000000000000
- 111111111111
Note that the 000000000000
account has two buckets dedicated to it. The first bucket (/AWSLogs/000000000000
) is empty. The real logs are at /AWSLogs/o-123/000000000000
. Looks like the code identifies the first bucket as being the one to make queries against, which it should use the one that is a child of the org key.
Document privileges needed by the users and for setting this up.
The recent PR that was merged enforced a \
on log_path_prefix. When that value is ``, that creates a problem.
Hi, when using AWS organisations the logfile path need to be changed. I added organisation id after AWSLogs and did a redeploy but I think the best is to let the config handle it.
209c209
< log_path_prefix = log_path_prefix + "AWSLogs/"
---
> log_path_prefix = log_path_prefix + "AWSLogs/[AWSORGANISATIONID]/"
If you have a large environment with many AWS accounts, you won't want to run this from your laptop as the initial setup will take hours. If you run this from an EC2, no default region is set. We should get the location of the S3 bucket, and then set the region to that.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.