joshlarsen / aws-recon Goto Github PK

View Code? Open in Web Editor NEW

516.0 16.0 49.0 274 KB

Multi-threaded AWS inventory collection tool with a focus on security-relevant resources and metadata.

Home Page: https://darkbit.io/resources

License: MIT License

Ruby 90.78% Shell 0.11% Dockerfile 0.62% HCL 8.49%

aws collection inventory security scanner cli

aws-recon's People

Contributors

Stargazers

Watchers

aws-recon's Issues

[EC2] describe_internet_gateways

Collect describe_internet_gateways. Needed for OpenCSPM aws-168 (Enterprise Control Pack).

[Redshift] describe_logging_status

Collect describe_logging_status. Needed for OpenCSPM aws-159 (Enterprise Control Pack).

CloudTrailARNInvalidException while ingesting CloudTrail

Hi, I'm getting the following CloudTrailARNInvalidException while ingesting CloudTrail:

root@docker:/app# AWS_PROFILE=test ./recon.rb -v

Starting collection with 8 threads...
t0.global.Organizations.describe_organization
t0.global.Organizations.list_handshakes_for_account.0
t0.global.EC2.describe_account_attributes
t0.global.IAM.get_account_authorization_details.0
t0.global.IAM.get_account_password_policy
t0.global.IAM.get_account_summary
t0.global.IAM.list_server_certificates.0
t0.global.IAM.list_virtual_mfa_devices.0
t0.global.IAM.ReportNotPresent
t0.global.S3.list_buckets.0
t0.global.S3.list_buckets.cloudseclist-test
t1.global.S3.list_buckets.csl-test-sest2.global.S3.list_buckets.ml-central-logging
t0.global.Route53Domains.list_domains.0
t0.global.Shield.ResourceNotFoundException
t0.global.Support.SubscriptionRequiredException
t6.eu-north-1.EC2.describe_instances.0
t5.eu-north-1.CloudTrail.describe_trails.0
t6.eu-north-1.EC2.describe_vpcs.0
t0.eu-north-1.CodeBuild.list_projects.0
t7.eu-north-1.EKS.list_clusters.0
t2.eu-north-1.CodePipeline.list_pipelines.0
t6.eu-north-1.EC2.describe_security_groups.0
t1.eu-north-1.ConfigService.describe_config_rules.0
t6.eu-north-1.EC2.describe_network_interfaces.0
t1.eu-north-1.ConfigService.describe_configuration_recorders.0
t1.eu-north-1.ConfigService.describe_delivery_channels.0
t6.eu-north-1.EC2.describe_subnets.0
t4.eu-north-1.AutoScaling.describe_auto_scaling_groups.0
t7.eu-north-1.ElasticLoadBalancing.describe_load_balancers.0
t0.eu-north-1.ECS.describe_clusters.0
t6.eu-north-1.EC2.describe_addresses.0
t6.eu-north-1.EC2.describe_nat_gateways.0
t6.eu-north-1.EC2.describe_route_tables.0
t3.eu-north-1.CloudFront.list_distributions.0
t6.eu-north-1.EC2.describe_images.0
t6.eu-north-1.EC2.describe_snapshots.0
t6.eu-north-1.EC2.describe_flow_logs.0
t6.eu-north-1.EC2.describe_volumes.0
t6.eu-north-1.EC2.describe_vpn_gateways.0
t6.eu-north-1.EC2.describe_vpc_peering_connections.0

Finished in 13 seconds. Saving resources to output.json.

Traceback (most recent call last):
	20: from /usr/local/bundle/gems/parallel-1.19.2/lib/parallel.rb:211:in `block (4 levels) in in_threads'
	19: from /usr/local/bundle/gems/parallel-1.19.2/lib/parallel.rb:360:in `block in work_in_threads'
	18: from /usr/local/bundle/gems/parallel-1.19.2/lib/parallel.rb:519:in `with_instrumentation'
	17: from /usr/local/bundle/gems/parallel-1.19.2/lib/parallel.rb:361:in `block (2 levels) in work_in_threads'
	16: from /usr/local/bundle/gems/parallel-1.19.2/lib/parallel.rb:508:in `call_with_index'
	15: from ./recon.rb:91:in `block (2 levels) in <main>'
	14: from ./recon.rb:44:in `collect'
	13: from /app/collectors/cloudtrail.rb:10:in `collect'
	12: from /app/collectors/cloudtrail.rb:10:in `each_with_index'
	11: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/aws-sdk-core/pageable_response.rb:91:in `each'
	10: from /app/collectors/cloudtrail.rb:13:in `block in collect'
	 9: from /app/collectors/cloudtrail.rb:13:in `each'
	 8: from /app/collectors/cloudtrail.rb:22:in `block (2 levels) in collect'
	 7: from /usr/local/bundle/gems/aws-sdk-cloudtrail-1.26.0/lib/aws-sdk-cloudtrail/client.rb:978:in `list_tags'
	 6: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/seahorse/client/request.rb:72:in `send_request'
	 5: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/seahorse/client/plugins/response_target.rb:24:in `call'
	 4: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
	 3: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
	 2: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
	 1: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': AccountID in ARN is wrong, you can only access resources with AccountID <REDACTED>. (Aws::CloudTrail::Errors::CloudTrailARNInvalidException)

[GuardDuty] list_findings

Collecting list_findings. Needed for OpenCSPM aws-177 (Enterprise Control Pack).

[CloudTrail] get_event_selectors

Get event selectors. Needed for OpenCSPM aws-120 and aws-121 Community Control Pack.

[SSM] describe_instance_patch_states_for_patch_group

Collect describe_instance_patch_states_for_patch_group for patch status. Needed for OpenCSPM aws-176 (Enterprise Control Pack).

Collect network interfaces from EC2 API

See describe_network_interfaces.

[S3] parse bucket policy

Unencode policy. Needed for OpenCSPM aws-192 (Enterprise Control Pack).

aws-recon not honoring AWS_PROFILE variable

I tried running aws-recon for multiple accounts. The instance where aws-recon resides is part of an account with a role attached to it and an aws config file with assume role profiles to multiple accounts.

The format of my ~/.aws/config looks like this :-

....

[profile  random-1 ]
role_arn =  arn:aws:iam::111111111111:role/randomRole
credential_source = Ec2InstanceMetadata

[profile  random-2 ]
role_arn =  arn:aws:iam::222222222222:role/randomRole
credential_source = Ec2InstanceMetadata

.....

This is the command I used

AWS_REGION=us-east-1 AWS_PROFILE=random-1 ruby recon.rb --services=s3 -v

The output remains the same across each run and I found that its been scanning the current account and has not been honoring the AWS_PROFILE.

I have followed the same method as suggested in the README. I am not quite sure if this is a bug or I am doing some mistake.

Also I would suggest to have --profile as an argument to the tool which would resemble aws-cli like syntax and would be easier to use and adopt.

[ECR] parse policy_text

Parse policy_text field. Needed for OpenCSPM aws-59 (Enterprise Control Pack).

Proposal to add Last Service Accessed Details to IAM Entities ( User / Role / Group /Policies)

The last service accessed details from access advisor helps find whether the provided permissions are utilised efficiently.
This would be good metadata to add to IAM entities which can be further analysed for over permissions.

[SecretsManager] list_secrets

Collect list_secrets. Needed for OpenCSPM aws-151 and aws-152 (Enterprise Control Pack).

[DynamoDB] describe_limits

Collect table limits describe_limits-instance_method. Needed for aws-133 (Enterprise Control Pack).

InvalidSignatureException for profiles with default resgion not set to us-east

Hello,
I've found the following error to occur if the default region of the AWS profile used is not set to us-east-1 (I've tried with eu-west-2 and us-west-1):

root@docker:/app# AWS_PROFILE=test ./recon.rb -v

Starting collection with 8 threads...

Finished in 0 seconds. Saving resources to output.json.

Traceback (most recent call last):
	12: from ./recon.rb:67:in `<main>'
	11: from ./recon.rb:67:in `each'
	10: from ./recon.rb:74:in `block in <main>'
	 9: from ./recon.rb:44:in `collect'
	 8: from /app/collectors/organizations.rb:11:in `collect'
	 7: from /usr/local/bundle/gems/aws-sdk-organizations-1.44.0/lib/aws-sdk-organizations/client.rb:2036:in `describe_organization'
	 6: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/seahorse/client/request.rb:72:in `send_request'
	 5: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/seahorse/client/plugins/response_target.rb:24:in `call'
	 4: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
	 3: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
	 2: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
	 1: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': Credential should be scoped to a valid region, not 'eu-west-2'.  (Aws::Organizations::Errors::InvalidSignatureException)

AWS profile:

[profile test]
output = json
region = eu-west-2

[EMR] list_clusters & describe_cluster

Collect list_clusters and describe_cluster. Needed for OpenCSPM aws-149 (Enterprise Control Pack).

Package as a gem

[Lambda] get_policy

Collect get_policy. Needed for OpenCSPM aws-170 (Enterprise Control Pack).

CloudTrail client using wrong region

CloudTrail calls fail if a prior trail required changing home_region.

Traceback (most recent call last):
	23: from ./recon.rb:80:in `<main>'
	22: from ./recon.rb:80:in `each'
	21: from ./recon.rb:81:in `block in <main>'
	20: from /.rvm/gems/ruby-2.6.5@aws-ruby-recon/gems/parallel-1.19.2/lib/parallel.rb:274:in `map'
	19: from /.rvm/gems/ruby-2.6.5@aws-ruby-recon/gems/parallel-1.19.2/lib/parallel.rb:336:in `work_direct'
	18: from /.rvm/gems/ruby-2.6.5@aws-ruby-recon/gems/parallel-1.19.2/lib/parallel.rb:519:in `with_instrumentation'
	17: from /.rvm/gems/ruby-2.6.5@aws-ruby-recon/gems/parallel-1.19.2/lib/parallel.rb:337:in `block in work_direct'
	16: from /.rvm/gems/ruby-2.6.5@aws-ruby-recon/gems/parallel-1.19.2/lib/parallel.rb:508:in `call_with_index'
	15: from ./recon.rb:91:in `block (2 levels) in <main>'
	14: from ./recon.rb:44:in `collect'
	13: from /code/tools/aws-recon/collectors/cloudtrail.rb:10:in `collect'
	12: from /code/tools/aws-recon/collectors/cloudtrail.rb:10:in `each_with_index'
	11: from /.rvm/gems/ruby-2.6.5@aws-ruby-recon/gems/aws-sdk-core-3.103.0/lib/aws-sdk-core/pageable_response.rb:91:in `each'
	10: from /code/tools/aws-recon/collectors/cloudtrail.rb:13:in `block in collect'
	 9: from /code/tools/aws-recon/collectors/cloudtrail.rb:13:in `each'
	 8: from /code/tools/aws-recon/collectors/cloudtrail.rb:21:in `block (2 levels) in collect'
	 7: from /.rvm/gems/ruby-2.6.5@aws-ruby-recon/gems/aws-sdk-cloudtrail-1.26.0/lib/aws-sdk-cloudtrail/client.rb:978:in `list_tags'
	 6: from /.rvm/gems/ruby-2.6.5@aws-ruby-recon/gems/aws-sdk-core-3.103.0/lib/seahorse/client/request.rb:72:in `send_request'
	 5: from /.rvm/gems/ruby-2.6.5@aws-ruby-recon/gems/aws-sdk-core-3.103.0/lib/seahorse/client/plugins/response_target.rb:24:in `call'
	 4: from /.rvm/gems/ruby-2.6.5@aws-ruby-recon/gems/aws-sdk-core-3.103.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
	 3: from /.rvm/gems/ruby-2.6.5@aws-ruby-recon/gems/aws-sdk-core-3.103.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
	 2: from /.rvm/gems/ruby-2.6.5@aws-ruby-recon/gems/aws-sdk-core-3.103.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
	 1: from /.rvm/gems/ruby-2.6.5@aws-ruby-recon/gems/aws-sdk-core-3.103.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'
/.rvm/gems/ruby-2.6.5@aws-ruby-recon/gems/aws-sdk-core-3.103.0/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': Region in ARN is wrong, you can only access resources in region us-east-1. (Aws::CloudTrail::Errors::CloudTrailARNInvalidException)

[SNS] parse policy documents

Parse Policy and EffectiveDeliveryPolicy fields.

Custom formatter for OpenCSPM

Rename custom formatter in lib/aws_recon/lib/formatter.rb to reflect OpenCSPM/opencspm format used by RedisGraph loader.

undefined method `downcase' for nil:NilClass (NoMethodError)

Was testing the tool on my Mac, and I get the following error:

Profile: default

$ ./recon.rb -s ec2
Traceback (most recent call last):
	8: from ./recon.rb:5:in `<main>'
	7: from /Users/<redacted>/Downloads/Tools/aws-recon/config/options.rb:135:in `parse'
	6: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/optparse.rb:1678:in `parse!'
	5: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/optparse.rb:1656:in `permute!'
	4: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/optparse.rb:1562:in `order!'
	3: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/optparse.rb:1568:in `parse_in_order'
	2: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/optparse.rb:1568:in `catch'
	1: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/optparse.rb:1614:in `block in parse_in_order'
/Users/<redacted>/Downloads/Tools/aws-recon/config/options.rb:65:in `block (2 levels) in parse': undefined method `downcase' for nil:NilClass (NoMethodError)

Tried the following as well:

$ AWS_PROFILE=<some_profile_name> ./recon.rb -s ec2
Traceback (most recent call last):
	8: from ./recon.rb:5:in `<main>'
	7: from /Users/<redacted>/Downloads/Tools/aws-recon/config/options.rb:135:in `parse'
	6: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/optparse.rb:1678:in `parse!'
	5: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/optparse.rb:1656:in `permute!'
	4: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/optparse.rb:1562:in `order!'
	3: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/optparse.rb:1568:in `parse_in_order'
	2: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/optparse.rb:1568:in `catch'
	1: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/optparse.rb:1614:in `block in parse_in_order'
/Users/<redacted>/Downloads/Tools/aws-recon/config/options.rb:65:in `block (2 levels) in parse': undefined method `downcase' for nil:NilClass (NoMethodError)

My ruby version:

$ ruby -v
ruby 2.6.3p62 (2019-04-16 revision 67580) [universal.x86_64-darwin19]

Any pointers for troubleshooting this?

[VPC] site-to-site tunnel status

Collect EC2.describe_vpn_connections. Needed for aws-135 (Enterprise Control Pack).

[SageMaker] list_endpoints & describe_endpoint

Collect list_endpoints and describe_endpoint. Needed for OpenCSPM aws-185 (Enterprise Control Pack).

AWS Premium Support Subscription is required to use this service. (Aws::Support::Errors::SubscriptionRequiredException)

Run log below :-

~/aws-recon #  AWS_PROFILE=default ruby recon.rb -v
Starting collection...
t3.global.Shield.ResourceNotFoundException
t2.global.S3.list_buckets.0
t2.global.S3.list_buckets.config-bucket-<redacted>
t0.global.EC2.describe_account_attributes
t2.global.S3.list_buckets.<redacted>
t2.global.S3.list_buckets.<redacted>
t1.global.IAM.get_account_authorization_details.0
t1.global.IAM.get_account_authorization_details.1
t1.global.IAM.get_account_password_policy
t1.global.IAM.get_account_summary
t1.global.IAM.list_server_certificates.0
t1.global.IAM.list_virtual_mfa_devices.0

Finished in 17 seconds. Saving resources to output.json.

Traceback (most recent call last):
	15: from /usr/local/bundle/gems/parallel-1.19.2/lib/parallel.rb:211:in `block (4 levels) in in_threads'
	14: from /usr/local/bundle/gems/parallel-1.19.2/lib/parallel.rb:360:in `block in work_in_threads'
	13: from /usr/local/bundle/gems/parallel-1.19.2/lib/parallel.rb:519:in `with_instrumentation'
	12: from /usr/local/bundle/gems/parallel-1.19.2/lib/parallel.rb:361:in `block (2 levels) in work_in_threads'
	11: from /usr/local/bundle/gems/parallel-1.19.2/lib/parallel.rb:508:in `call_with_index'
	10: from recon.rb:72:in `block in <main>'
	 9: from recon.rb:42:in `collect'
	 8: from /root/aws-recon/collectors/support.rb:11:in `collect'
	 7: from /usr/local/bundle/gems/aws-sdk-support-1.25.0/lib/aws-sdk-support/client.rb:1219:in `describe_trusted_advisor_checks'
	 6: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/seahorse/client/request.rb:72:in `send_request'
	 5: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/seahorse/client/plugins/response_target.rb:24:in `call'
	 4: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
	 3: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
	 2: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
	 1: from /usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.103.0/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': AWS Premium Support Subscription is required to use this service. (Aws::Support::Errors::SubscriptionRequiredException)

These are my environment details :-

Running it in Docker Image : ruby:2.6-alpine

Dockerfile below :-

FROM ruby2.6:alpine
WORKDIR /aws-recon/
apk update && apk --no-cache --update add git build-base
bundler install

I have cloned the repo and mounted the folder within /aws-recon/ path inside the container.

Docker Container OS details:

~/aws-recon #  uname -a
Linux 51c29d7be40c 5.4.0-40-generic #44-Ubuntu SMP Tue Jun 23 00:01:04 UTC 2020 x86_64 Linux

Local Ruby Gems installed :-

~/aws-recon # gem query --local

*** LOCAL GEMS ***

ast (2.4.1)
aws-eventstream (1.1.0)
aws-partitions (1.339.0)
aws-sdk (3.0.1)
aws-sdk-accessanalyzer (1.9.0)
aws-sdk-acm (1.34.0)
aws-sdk-acmpca (1.26.0)
aws-sdk-alexaforbusiness (1.39.0)
aws-sdk-amplify (1.20.0)
aws-sdk-apigateway (1.48.0)
aws-sdk-apigatewaymanagementapi (1.16.0)
aws-sdk-apigatewayv2 (1.23.0)
aws-sdk-appconfig (1.9.0)
aws-sdk-applicationautoscaling (1.43.0)
aws-sdk-applicationdiscoveryservice (1.30.0)
aws-sdk-applicationinsights (1.12.0)
aws-sdk-appmesh (1.27.0)
aws-sdk-appstream (1.44.0)
aws-sdk-appsync (1.29.0)
aws-sdk-athena (1.30.0)
aws-sdk-augmentedairuntime (1.7.0)
aws-sdk-autoscaling (1.43.0)
aws-sdk-autoscalingplans (1.25.0)
aws-sdk-backup (1.18.0)
aws-sdk-batch (1.34.0)
aws-sdk-budgets (1.32.0)
aws-sdk-chime (1.32.0)
aws-sdk-cloud9 (1.25.0)
aws-sdk-clouddirectory (1.26.0)
aws-sdk-cloudformation (1.41.0)
aws-sdk-cloudfront (1.34.0)
aws-sdk-cloudhsm (1.24.0)
aws-sdk-cloudhsmv2 (1.26.0)
aws-sdk-cloudsearch (1.23.0)
aws-sdk-cloudsearchdomain (1.19.0)
aws-sdk-cloudtrail (1.26.0)
aws-sdk-cloudwatch (1.41.0)
aws-sdk-cloudwatchevents (1.32.0)
aws-sdk-cloudwatchlogs (1.34.0)
aws-sdk-codeartifact (1.1.0)
aws-sdk-codebuild (1.56.0)
aws-sdk-codecommit (1.37.0)
aws-sdk-codedeploy (1.34.0)
aws-sdk-codeguruprofiler (1.8.0)
aws-sdk-codegurureviewer (1.9.0)
aws-sdk-codepipeline (1.34.0)
aws-sdk-codestar (1.24.0)
aws-sdk-codestarconnections (1.7.0)
aws-sdk-codestarnotifications (1.5.0)
aws-sdk-cognitoidentity (1.24.0)
aws-sdk-cognitoidentityprovider (1.42.0)
aws-sdk-cognitosync (1.21.0)
aws-sdk-comprehend (1.34.0)
aws-sdk-comprehendmedical (1.20.0)
aws-sdk-computeoptimizer (1.5.0)
aws-sdk-configservice (1.48.0)
aws-sdk-connect (1.28.0)
aws-sdk-connectparticipant (1.5.0)
aws-sdk-core (3.103.0)
aws-sdk-costandusagereportservice (1.24.0)
aws-sdk-costexplorer (1.44.0)
aws-sdk-databasemigrationservice (1.38.0)
aws-sdk-dataexchange (1.7.0)
aws-sdk-datapipeline (1.21.0)
aws-sdk-datasync (1.22.0)
aws-sdk-dax (1.24.0)
aws-sdk-detective (1.8.0)
aws-sdk-devicefarm (1.36.0)
aws-sdk-directconnect (1.33.0)
aws-sdk-directoryservice (1.31.0)
aws-sdk-dlm (1.31.0)
aws-sdk-docdb (1.20.0)
aws-sdk-dynamodb (1.51.0)
aws-sdk-dynamodbstreams (1.21.0)
aws-sdk-ebs (1.6.0)
aws-sdk-ec2 (1.175.0)
aws-sdk-ec2instanceconnect (1.8.0)
aws-sdk-ecr (1.34.0)
aws-sdk-ecs (1.67.0)
aws-sdk-efs (1.33.0)
aws-sdk-eks (1.39.0)
aws-sdk-elasticache (1.40.0)
aws-sdk-elasticbeanstalk (1.34.0)
aws-sdk-elasticinference (1.7.0)
aws-sdk-elasticloadbalancing (1.25.0)
aws-sdk-elasticloadbalancingv2 (1.47.0)
aws-sdk-elasticsearchservice (1.39.0)
aws-sdk-elastictranscoder (1.24.0)
aws-sdk-emr (1.34.0)
aws-sdk-eventbridge (1.10.0)
aws-sdk-firehose (1.31.0)
aws-sdk-fms (1.28.0)
aws-sdk-forecastqueryservice (1.7.0)
aws-sdk-forecastservice (1.7.0)
aws-sdk-frauddetector (1.6.0)
aws-sdk-fsx (1.23.0)
aws-sdk-gamelift (1.34.0)
aws-sdk-glacier (1.32.0)
aws-sdk-globalaccelerator (1.20.0)
aws-sdk-glue (1.63.0)
aws-sdk-greengrass (1.33.0)
aws-sdk-groundstation (1.10.0)
aws-sdk-guardduty (1.36.0)
aws-sdk-health (1.27.0)
aws-sdk-honeycode (1.0.0)
aws-sdk-iam (1.43.0)
aws-sdk-imagebuilder (1.11.0)
aws-sdk-importexport (1.21.0)
aws-sdk-inspector (1.29.0)
aws-sdk-iot (1.54.0)
aws-sdk-iot1clickdevicesservice (1.23.0)
aws-sdk-iot1clickprojects (1.23.0)
aws-sdk-iotanalytics (1.31.0)
aws-sdk-iotdataplane (1.23.0)
aws-sdk-iotevents (1.17.0)
aws-sdk-ioteventsdata (1.10.0)
aws-sdk-iotjobsdataplane (1.22.0)
aws-sdk-iotsecuretunneling (1.5.0)
aws-sdk-iotsitewise (1.7.0)
aws-sdk-iotthingsgraph (1.9.0)
aws-sdk-kafka (1.23.0)
aws-sdk-kendra (1.8.0)
aws-sdk-kinesis (1.26.0)
aws-sdk-kinesisanalytics (1.26.0)
aws-sdk-kinesisanalyticsv2 (1.18.0)
aws-sdk-kinesisvideo (1.27.0)
aws-sdk-kinesisvideoarchivedmedia (1.26.0)
aws-sdk-kinesisvideomedia (1.23.0)
aws-sdk-kinesisvideosignalingchannels (1.5.0)
aws-sdk-kms (1.36.0)
aws-sdk-lakeformation (1.7.0)
aws-sdk-lambda (1.46.0)
aws-sdk-lambdapreview (1.21.0)
aws-sdk-lex (1.28.0)
aws-sdk-lexmodelbuildingservice (1.33.0)
aws-sdk-licensemanager (1.16.0)
aws-sdk-lightsail (1.35.0)
aws-sdk-machinelearning (1.22.0)
aws-sdk-macie (1.22.0)
aws-sdk-macie2 (1.5.0)
aws-sdk-managedblockchain (1.13.0)
aws-sdk-marketplacecatalog (1.5.0)
aws-sdk-marketplacecommerceanalytics (1.26.0)
aws-sdk-marketplaceentitlementservice (1.21.0)
aws-sdk-marketplacemetering (1.28.0)
aws-sdk-mediaconnect (1.24.0)
aws-sdk-mediaconvert (1.53.0)
aws-sdk-medialive (1.49.0)
aws-sdk-mediapackage (1.30.0)
aws-sdk-mediapackagevod (1.16.0)
aws-sdk-mediastore (1.27.0)
aws-sdk-mediastoredata (1.24.0)
aws-sdk-mediatailor (1.29.0)
aws-sdk-migrationhub (1.26.0)
aws-sdk-migrationhubconfig (1.6.0)
aws-sdk-mobile (1.21.0)
aws-sdk-mq (1.29.0)
aws-sdk-mturk (1.24.0)
aws-sdk-neptune (1.27.0)
aws-sdk-networkmanager (1.5.0)
aws-sdk-opsworks (1.27.0)
aws-sdk-opsworkscm (1.37.0)
aws-sdk-organizations (1.44.0)
aws-sdk-outposts (1.7.0)
aws-sdk-personalize (1.15.0)
aws-sdk-personalizeevents (1.9.0)
aws-sdk-personalizeruntime (1.13.0)
aws-sdk-pi (1.21.0)
aws-sdk-pinpoint (1.43.0)
aws-sdk-pinpointemail (1.21.0)
aws-sdk-pinpointsmsvoice (1.18.0)
aws-sdk-polly (1.34.0)
aws-sdk-pricing (1.21.0)
aws-sdk-qldb (1.8.0)
aws-sdk-qldbsession (1.6.0)
aws-sdk-quicksight (1.25.0)
aws-sdk-ram (1.19.0)
aws-sdk-rds (1.93.0)
aws-sdk-rdsdataservice (1.20.0)
aws-sdk-redshift (1.46.0)
aws-sdk-rekognition (1.42.0)
aws-sdk-resourcegroups (1.26.0)
aws-sdk-resourcegroupstaggingapi (1.28.0)
aws-sdk-resources (3.75.0)
aws-sdk-robomaker (1.26.0)
aws-sdk-route53 (1.40.0)
aws-sdk-route53domains (1.25.0)
aws-sdk-route53resolver (1.17.0)
aws-sdk-s3 (1.73.0)
aws-sdk-s3control (1.21.0)
aws-sdk-sagemaker (1.62.0)
aws-sdk-sagemakerruntime (1.24.0)
aws-sdk-savingsplans (1.7.0)
aws-sdk-schemas (1.6.0)
aws-sdk-secretsmanager (1.39.0)
aws-sdk-securityhub (1.29.0)
aws-sdk-serverlessapplicationrepository (1.29.0)
aws-sdk-servicecatalog (1.43.0)
aws-sdk-servicediscovery (1.26.0)
aws-sdk-servicequotas (1.8.0)
aws-sdk-ses (1.33.0)
aws-sdk-sesv2 (1.8.0)
aws-sdk-shield (1.29.0)
aws-sdk-signer (1.23.0)
aws-sdk-simpledb (1.21.0)
aws-sdk-sms (1.23.0)
aws-sdk-snowball (1.31.0)
aws-sdk-sns (1.27.0)
aws-sdk-sqs (1.30.0)
aws-sdk-ssm (1.84.0)
aws-sdk-sso (1.6.0)
aws-sdk-ssooidc (1.5.0)
aws-sdk-states (1.31.0)
aws-sdk-storagegateway (1.45.0)
aws-sdk-support (1.25.0)
aws-sdk-swf (1.22.0)
aws-sdk-synthetics (1.5.0)
aws-sdk-textract (1.17.0)
aws-sdk-transcribeservice (1.45.0)
aws-sdk-transcribestreamingservice (1.17.0)
aws-sdk-transfer (1.23.0)
aws-sdk-translate (1.24.0)
aws-sdk-waf (1.33.0)
aws-sdk-wafregional (1.34.0)
aws-sdk-wafv2 (1.8.0)
aws-sdk-workdocs (1.25.0)
aws-sdk-worklink (1.18.0)
aws-sdk-workmail (1.27.0)
aws-sdk-workmailmessageflow (1.6.0)
aws-sdk-workspaces (1.40.0)
aws-sdk-xray (1.28.0)
aws-sigv2 (1.0.1)
aws-sigv4 (1.2.1)
backport (1.1.2)
benchmark (0.1.0)
bigdecimal (default: 1.4.1)
bundler (default: 1.17.2)
cmath (default: 1.0.0)
coderay (1.1.3)
csv (default: 3.0.9)
date (default: 2.0.0)
dbm (default: 1.0.0)
did_you_mean (1.3.0)
e2mmap (default: 0.1.0)
etc (default: 1.0.1)
fcntl (default: 1.0.0)
fiddle (default: 1.0.0)
fileutils (default: 1.1.0)
forwardable (default: 1.2.0)
gdbm (default: 2.0.0)
io-console (default: 0.4.7)
ipaddr (default: 1.2.2)
irb (default: 1.0.0)
jaro_winkler (1.5.4)
jmespath (1.4.0)
json (default: 2.1.0)
logger (default: 1.3.0)
maruku (0.7.3)
matrix (default: 0.1.0)
method_source (1.0.0)
mini_portile2 (2.4.0)
minitest (5.11.3)
mutex_m (default: 0.1.0)
net-telnet (0.2.0)
nokogiri (1.10.10)
openssl (default: 2.1.2)
ostruct (default: 0.1.0)
parallel (1.19.2)
parser (2.7.1.4)
power_assert (1.1.3)
prime (default: 0.1.0)
pry (0.13.1)
psych (default: 3.1.0)
rainbow (3.0.0)
rake (12.3.3)
rdoc (default: 6.1.2)
regexp_parser (1.7.1)
reverse_markdown (2.0.0)
rexml (3.2.4, default: 3.1.9)
rss (default: 0.2.7)
rubocop (0.87.1)
rubocop-ast (0.1.0)
ruby-progressbar (1.10.1)
scanf (default: 1.0.0)
sdbm (default: 1.0.0)
shell (default: 0.7)
solargraph (0.39.11)
stringio (default: 0.0.2)
strscan (default: 1.0.0)
sync (default: 0.5.0)
test-unit (3.2.9)
thor (1.0.1)
thwait (default: 0.1.0)
tilt (2.0.10)
tracer (default: 0.1.0)
unicode-display_width (1.7.0)
webrick (default: 1.4.2)
xmlrpc (0.3.0)
yard (0.9.25)
zlib (default: 1.0.0)

Let me know if you need any more details

[DynamoDB] auto_scaling_plans

Collect describe_table_replica_auto_scaling. Needed for aws-136 (Enterprise Control Pack)

`user_policy_list.policy_document` is URL encoded

When downloading IAM data, the policy_document for each record nested under user_policy_list appears to be URL encoded. I was able to use URI.decode_www_form on the string to turn it into the actual policy document.

Re-raise exceptions

Change default behavior to always suppress AccessDenied exceptions. Add command line option to re-raise for troubleshooting.

[AccessAnalyzer] list_analyzers

Messy policy document for IAM service

Hello,
produced via ./recon.rb -v --services IAM -r global - JSON policy documents seems messy:

commit a861691

ruby --version
ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]

Resources Reordering Suggestion

Propose moving:

Route53 Zone list-query-logging-configs as a sub resource to the zone
VPC describe_flow_logs under describe_vpc

[SecurityHub] describe_hub

Collect describe_hub. Needed for OpenCSPM aws-160 (Enterprise Control Pack).

[S3] get_bucket_replication

Collection bucket replication settings get_bucket_replication. Needed for OpenCSPM aws-146 (Enterprise Control Pack).

[Backup] collect backup protected resources

Needed for aws-139 (Enterprise Control Pack).

[docker] GitHub Action to push gem and build Docker image

Trigger Action on push to main with lib/aws_recon/version.rb changes.

release gem in pre-commit hook
build Docker image
push to Docker Hub

[APIGateway] get_authorizers

get_authorizers.

Needed for OpenCSPM aws-79.

KMS key enrichment calls get skipped

In kms.rb, if get_key_rotation_status can't be called, an exception is raised and the remaining enrichment calls are skipped. This means the key's grants, policies, and aliases won't be collected.

[SageMaker] describe_notebook_instance

Collect describe_notebook_instance for direct_internet_access and kms_key_id. Needed for OpenCSPM aws-174 and aws-186 (Enterprise Control Pack).

[IAM] list_policies

List all managed policies

[CloudTrail] AccountID in ARN is wrong error when used with assumed roles

Hi.
I have encouraged issues with IAM and CloudTrail when used with Assumed role in to another account, with full readonly permission. (AWS builtIn ReadOnly policy)
Here tracebacks.
I had to exclude IAM and CloudTrail from collection.

Traceback (most recent call last):
	21: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/parallel-1.19.2/lib/parallel.rb:211:in 'block (4 levels) in in_threads'
	20: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/parallel-1.19.2/lib/parallel.rb:360:in 'block in work_in_threads'
	19: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/parallel-1.19.2/lib/parallel.rb:519:in 'with_instrumentation'
	18: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/parallel-1.19.2/lib/parallel.rb:361:in 'block (2 levels) in work_in_threads'
	17: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/parallel-1.19.2/lib/parallel.rb:508:in 'call_with_index'
	16: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/aws_recon-0.2.7/lib/aws_recon/aws_recon.rb:99:in 'block (2 levels) in start'
	15: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/aws_recon-0.2.7/lib/aws_recon/aws_recon.rb:49:in 'collect'
	14: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/aws_recon-0.2.7/lib/aws_recon/collectors/cloudtrail.rb:10:in 'collect'
	13: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/aws_recon-0.2.7/lib/aws_recon/collectors/cloudtrail.rb:10:in 'each_with_index'
	12: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/aws-sdk-core-3.109.1/lib/aws-sdk-core/pageable_response.rb:93:in 'each'
	11: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/aws_recon-0.2.7/lib/aws_recon/collectors/cloudtrail.rb:13:in 'block in collect'
	10: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/aws_recon-0.2.7/lib/aws_recon/collectors/cloudtrail.rb:13:in 'each'
	 9: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/aws_recon-0.2.7/lib/aws_recon/collectors/cloudtrail.rb:22:in 'block (2 levels) in collect'
	 8: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/aws-sdk-cloudtrail-1.29.0/lib/aws-sdk-cloudtrail/client.rb:993:in 'list_tags'
	 7: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/aws-sdk-core-3.109.1/lib/seahorse/client/request.rb:72:in 'send_request'
	 6: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/aws-sdk-core-3.109.1/lib/seahorse/client/plugins/response_target.rb:24:in 'call'
	 5: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/aws-sdk-core-3.109.1/lib/aws-sdk-core/plugins/response_paging.rb:12:in 'call'
	 4: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/aws-sdk-core-3.109.1/lib/seahorse/client/plugins/request_callback.rb:71:in 'call'
	 3: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/aws-sdk-core-3.109.1/lib/aws-sdk-core/plugins/param_converter.rb:26:in 'call'
	 2: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/aws-sdk-core-3.109.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in 'call'
	 1: from /Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/aws-sdk-core-3.109.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in 'call'
/Users/{USERNAME}/.rvm/gems/ruby-2.6.5/gems/aws-sdk-core-3.109.1/lib/seahorse/client/plugins/raise_response_errors.rb:17:in 'call': AccountID in ARN is wrong, you can only access resources with AccountID {AccountNumber}. (Aws::CloudTrail::Errors::CloudTrailARNInvalidException)

Generate IAM reports

Several IAM reports generate useful metadata about credential and service usage:

Each of these report require generating the report (and waiting) prior to accessing the details. The current plan is to implement these using the SDK waiters. Since this will slow down considerably, it will likely be a separate command line option that is disabled by default.

[EC2] describe_network_acls

Collect decsribe_network_acls. Needed for OpenCSPM aws-122 Community Control Pack.

[SecurityHub] InvalidAccessException when account is not subscribed

Would it be possible to add feature, to proceed on some errors, for example if no services enabled, but still be able to collect other information?
/Users/user/.rvm/gems/ruby-2.6.5/gems/aws-sdk-core-3.109.1/lib/seahorse/client/plugins/raise_response_errors.rb:17:in 'call': Account 123123123123 is not subscribed to AWS Security Hub (Aws::SecurityHub::Errors::InvalidAccessException)

joshlarsen / aws-recon Goto Github PK

aws-recon's People

Contributors

Stargazers

Watchers

Forkers

aws-recon's Issues

Running it in Docker Image : ruby:2.6-alpine

Docker Container OS details:

Local Ruby Gems installed :-

Recommend Projects

Recommend Topics

Recommend Org