Use SQL to instantly query AWS resources across regions and accounts. Open source CLI. No DB required.
Home Page: https://hub.steampipe.io/plugins/turbot/aws
License: Apache License 2.0
What version of Steampipe are you using?
0.1.0
What operating system version are you using?
macOS 10.15.7 (Catalina)(Intel)
What plugins do you have installed?
aws, azure, steampipe
Describe what you are trying to do
When I run the query:
select name from aws_s3_bucket;I get the error:
Error: pq: rpc error: code = Internal desc = list call listS3Buckets failed with panic AWS_REGION must be set to use the aws extension
Expected Behavior
The query should return a list of buckets
To Reproduce
Execute following command:
steampipe query "select name from aws_s3_bucket;"
Thank you for reading!
aws_iam_account_password_policy
aws_iam_account_summary
aws_iam_action
aws_iam_credential_report
aws_iam_policy_simulator
We can use these tables to perform a variety of IAM checks - what wildcard-expanded actions do policies grant, can people perform a particular action, do users have passwords enabled, what unused access keys are there, is MFA enabled for users, answer large parts of CIS v1.1 etc.
Jut wanted to say thank you for putting this tool out there. It's been very helpful already for my day to day work trying to make sense of a sprawling / legacy TF set up.
I'd be grateful if route53 zones and records were added. I'm trying to get a basic list of the different zones in an account and a list of the records per zone in an account.
Something like:
select
domain_name as name,
type,
zone_id
from
aws_route53_zones;and
select
record_name as name,
type,
value,
zone_id
from
aws_route53_records INNER JOIN
aws_route53_zones ...
select account_id, region,title from aws_morales.aws_ebs_snapshot
Error: pq: rpc error: code = Unknown desc = AuthFailure: AWS was not able to validate the provided access credentials
status code: 401, request id: 868174ba-46a8-437d-8c6b-64ee448dd0dcallowing 'arn' as well as 'name' to be used as key columns will optimize joining tables like aws_iam_role, aws_iam_user, aws_iam_group, aws_iam_policy
This would be a breaking change, but I think it should be done ASAP as a bug fix / patch to make it aligned.
~/src/steampipe-plugin-aws $ grep -r tag_ aws
aws/table_aws_rds_db_cluster_snapshot.go: Name: "tag_list",
aws/table_aws_rds_db_option_group.go: Name: "tag_list",
aws/table_aws_rds_db_snapshot.go: Name: "tag_list",
aws/table_aws_rds_db_parameter_group.go: Name: "tag_list",
aws/table_aws_rds_db_subnet_group.go: Name: "tag_list",
aws/table_aws_rds_db_cluster.go: Name: "tag_list",
aws/table_aws_rds_db_cluster_parameter_group.go: Name: "tag_list",
aws/table_aws_rds_db_instance.go: Name: "tag_list",
~/src/steampipe-plugin-aws $
~/src/steampipe-plugin-aws $
~/src/steampipe-plugin-aws $
~/src/steampipe-plugin-aws $ grep -r tags_ aws
aws/table_aws_vpc_egress_only_internet_gateway.go: Name: "tags_src",
aws/table_aws_ec2_network_interface.go: Name: "tags_src",
aws/table_aws_kms_key.go: Name: "tags_src",
aws/table_aws_api_gateway_api_key.go: Name: "tags_src",
aws/table_aws_vpc_route_table.go: Name: "tags_src",
aws/table_aws_ec2_target_group.go: Name: "tags_src",
aws/table_aws_vpc_eip.go: Name: "tags_src",
aws/table_aws_ssm_parameter.go: Name: "tags_src",
aws/table_aws_vpc_security_group.go: Name: "tags_src",
aws/table_aws_cloudformation_stack.go: Name: "tags_src",
aws/table_aws_ec2_transit_gateway_route_table.go: Name: "tags_src",
aws/table_aws_vpc_internet_gateway.go: Name: "tags_src",
aws/table_aws_ec2_instance.go: Name: "tags_src",
aws/table_aws_s3_bucket.go: Name: "tags_src",
aws/table_aws_route53_zone.go: Name: "tags_src",
aws/table_aws_vpc_endpoint.go: Name: "tags_src",
aws/table_aws_vpc_subnet.go: Name: "tags_src",
aws/table_aws_ec2_key_pair.go: Name: "tags_src",
aws/table_aws_ec2_ami.go: Name: "tags_src",
aws/table_aws_rds_db_cluster.go: Name: "copy_tags_to_snapshot",
aws/table_aws_iam_user.go: Name: "tags_src",
aws/table_aws_vpc.go: Name: "tags_src",
aws/table_aws_ec2_transit_gateway_vpc_attachment.go: Name: "tags_src",
aws/table_aws_ebs_volume.go: Name: "tags_src",
aws/table_aws_acm_certificate.go: Name: "tags_src",
aws/table_aws_ec2_application_load_balancer.go: Name: "tags_src",
aws/table_aws_ec2_classic_load_balancer.go: Name: "tags_src",
aws/table_aws_vpc_dhcp_options.go: Name: "tags_src",
aws/table_aws_vpc_nat_gateway.go: Name: "tags_src",
aws/table_aws_ec2_transit_gateway.go: Name: "tags_src",
aws/table_aws_vpc_network_acl.go: Name: "tags_src",
aws/table_aws_ec2_network_load_balancer.go: Name: "tags_src",
aws/table_aws_dynamodb_table.go: Name: "tags_src",
aws/table_aws_vpc_vpn_gateway.go: Name: "tags_src",
aws/table_aws_vpc_endpoint_service.go: Name: "tags_src",
aws/table_aws_rds_db_instance.go: Name: "copy_tags_to_snapshot",
aws/table_aws_vpc_customer_gateway.go: Name: "tags_src",
aws/table_aws_sns_topic.go: Name: "tags_src",
aws/table_aws_ec2_autoscaling_group.go: Name: "tags_src",
aws/table_aws_iam_role.go: Name: "tags_src",
aws/table_aws_ebs_snapshot.go: Name: "tags_src",
~/src/steampipe-plugin-aws $
Below command breaks, when I query single bucket name.
Error: pq: rpc error: code = Internal desc = get hydrate function getS3Bucket failed with panic interface conversion: interface {} is nil, not *s3.Bucket
select name, jsonb_pretty(acl) as acl from aws_s3_bucket where name='kolkata-user-grp-demo';
Error: pq: rpc error: code = Internal desc = get hydrate function getS3Bucket failed with panic interface conversion: interface {} is nil, not *s3.Bucket
select name, jsonb_pretty(acl) as acl from aws_s3_bucket where name like '%kolkata%';
+-----------------------+-------------------------------------------------------------------------------------------+
| name | acl |
+-----------------------+-------------------------------------------------------------------------------------------+
| kolkata-user-grp-demo | {
. . . . . . . . | |
+-----------------------+-------------------------------------------------------------------------------------------+
Select * from aws_s3_bucket usually works without issue, but occasionally fails with error: Error: pq: rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated., an other times returns partial results.
Using steampipe version 0.2.1, aws plugin 0.5.1, caching (STEAMPIPE_CACHE) is OFF
Hi team
Would like to see AWS Config table support.
I get the following error when AWS_REGION is not set, even if I have set the regions argument in my connection configuration file:
Error: pq: rpc error: code = Internal desc = hydrate call getOrganizationDetails failed with panic
You must specify a region. You can configure your region by running "aws configure" or setting AWS_REGION environment variable.
The docs for the aws plugin indicate that in the default case, credentials are sourced using the standard credential resolution order. My understanding of steampipe's "connections" functionality is that I should be able to have multiple schemas for the AWS plugin by defining multiple connections.
I created a file at ~/.steampipe/config/test.spc with the contents:
connection "awstest" {
plugin = "aws"
profile = "profilename"
}I expected this would use the credentials for the profilename profile when I run the query select * from awstest.aws_account but it still used the default credentials. If I run AWS_PROFILE=profilename steampipe query ... then it works correctly, but then I can only access that account.
Is my assumption correct that I should be able to access different accounts using different connections? Has this functionality been written yet? When I look at the code for configuring the AWS session, it looks like maybe it hasn't been written yet:
steampipe-plugin-aws/aws/service.go
Line 456 in bc9afca
if regions argument is not specified for an aws connection in the config file, steampipe should use whatever region the aws CLI would use - either as set in the env vars or for the default profile. Currently, it errors even if AWS_REGION is set:
Welcome to Steampipe v0.1.3
For more information, type .help
> select
i.instance_id,
vols -> 'Ebs' ->> 'VolumeId' as vol_id,
vol.encrypted
from
aws_ec2_instance as i,
jsonb_array_elements(block_device_mappings) as vols
join aws_ebs_volume as vol on vol.volume_id = vols -> 'Ebs' ->> 'VolumeId'
Error: pq: rpc error: code = Unknown desc =
'regions' must be set in the connection configuration. Edit your connection configuration file and then restart Steampipe
> .quit
$ printenv |grep -i AWS_REGION
AWS_REGION=us-east-1
This table can be joined with the user table to get details for virtual MFA for each user
https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstanceStatus.html
Instance status includes the following components:
Status checks - Amazon EC2 performs status checks on running EC2 instances to identify hardware and software issues. For more information, see Status checks for your instances and Troubleshooting instances with failed status checks in the Amazon EC2 User Guide.
Scheduled events - Amazon EC2 can schedule events (such as reboot, stop, or terminate) for your instances related to hardware issues, software updates, or system maintenance. For more information, see Scheduled events for your instances in the Amazon EC2 User Guide.
Instance state - You can manage your instances from the moment you launch them through their termination. For more information, see Instance lifecycle in the Amazon EC2 User Guide.
in aws_iam_credential_report, we are converting N/A and no_information to <null> becuase they are timestamps.
Because of this, we lose some valuable information: password_last_used will indicate 'N/A' if the user does not have a console credential, but 'no_information' if the password is created but never used ( this is NOT true for keys however - access_key_1_last_used_date will be 'N/A' in both similar cases). We should add a bool column password_never_used and set it to true if password_last_used is set to 'no_information'. (column name up for discussion).
password_status with values of never_used, used, not_set.
https://twitter.com/ben11kehoe/status/1354907439749951488?s=21
The AWS #golang SDK has support for AWS SSO credentials in version 1.37! So you can set up a profile for AWS SSO, use aws sso login, set the env var AWS_SDK_LOAD_CONFIG=1, and then your Go program will correctly get creds for that profile!
https://github.com/aws/aws-sdk-go/blob/master/CHANGELOG.md#sdk-features
For this query:
select name, jsonb_pretty(lifecycle_rules) from aws_s3_bucket;
the current result is this:
| example-bucket | { |
| | "Rules": [ |
| | { |
| | "ID": "Delete after 90 days: AWSLogs/*", |
| | "Filter": null, |
| | "Prefix": "AWSLogs/", |
| | "Status": "Enabled", |
| | "Expiration": { |
| | "Date": null, |
| | "Days": 90, |
| | "ExpiredObjectDeleteMarker": null |
| | }, |
| | "Transitions": null, |
| | "NoncurrentVersionExpiration": { |
| | "NoncurrentDays": 90 |
| | }, |
| | "NoncurrentVersionTransitions": null, |
| | "AbortIncompleteMultipartUpload": null |
| | }, |
| | { |
| | "ID": "Delete after 90 days: TurbotLogs/Backups/*", |
| | "Filter": null, |
| | "Prefix": "TurbotLogs/Backups/", |
| | "Status": "Enabled", |
| | "Expiration": { |
| | "Date": null, |
| | "Days": 90, |
| | "ExpiredObjectDeleteMarker": null |
| | }, |
| | "Transitions": null, |
| | "NoncurrentVersionExpiration": { |
| | "NoncurrentDays": 90 |
| | }, |
| | "NoncurrentVersionTransitions": null, |
| | "AbortIncompleteMultipartUpload": null |
| | }, |
| | { |
| | "ID": "Delete after 90 days: TurbotLogs/CloudFormation/*", |
| | "Filter": null, |
| | "Prefix": "TurbotLogs/CloudFormation/", |
| | "Status": "Enabled", |
| | "Expiration": { |
| | "Date": null, |
| | "Days": 90, |
| | "ExpiredObjectDeleteMarker": null |
| | }, |
| | "Transitions": null, |
| | "NoncurrentVersionExpiration": { |
| | "NoncurrentDays": 90 |
| | }, |
| | "NoncurrentVersionTransitions": null, |
| | "AbortIncompleteMultipartUpload": null |
| | }, |
It should be an array [...] of rules, not an object {"Rules":[...]}.
We can use this table to join onto permission policies and provide a view of the level of access granted e.g. admin, readonly etc.
potentially, cost explorer, pricing api, etc
Currently the bucket is publicly accessible
aws s3api get-bucket-policy-status --bucket testrouterbucket04092020
but the query output showing that the bucket is not publicly accessible
Checklist:
title, tags, akas, are includedOver on Reddit, u/krad7958 asked about adding SSM parameter store support - https://www.reddit.com/r/aws/comments/l4e0bo/anyone_tried_steampipe_sql_querying_of_aws_yet/gkp3fi7
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.