janiko71 / aws-inventory Goto Github PK

Not a big issue, but I wonder if anyone has a quick solution for it.
I am using this script to generate an inventory for documentation purposes (I save nicely formatted yaml files rather than jsons). I wish to achieve two things

1. avoid adding empty sections as per below - is there a way to omit those?

ec2: []
ec2-ebs: []
ec2-egpu: []
ec2-eips: []

2. only return specific fields from an inventory. e.g:
   for EvenBus, I only need 'Name', not a whole blob of 17 lines.

Kind regards
Dawid

Returns an (incorrrect) empty list.

How can I limit the scan to a specific region ?

I’ve edited the aws-regions.json configuration to the us-east-1 region alone. However when I run a scan for s3 buckets the tool defaults to global region

Request advise

Hi

I don't know If this is an issue, but output always is json, and Im interested to use other options like text or table. I config inside AWS cli this option, but I can't see If exist any form to pass this option to the script.

Thanks

Hi
I'm checking script with parameters profile and services how appears in the wiki but appears error for parameters.

Exception: Unknown argument [--profile]
Exception: Unknown argument [--services]

simply I execute

python3 inventory.py --profile tools --services ec2

Which config is missing?

SSL validation failed for https://s3.amazonaws.com/ TLS/SSL connection has been closed (EOF) (_ssl.c:997)
2023-02-03 09:17:51,801 ERROR Error while processing elasticache, ap-south-1, describe_cache_clusters. Error: SSL validation failed for https://elasticache.ap-south-1.amazonaws.com/ TLS/SSL connection has been closed (EOF) (_ssl.c:997

since config.global_inventory list is empty so getting below error

Error :
Traceback (most recent call last):
File "/var/task/call_aws_inventory.py", line 5, in get_aws_inventory
inventory.call_inventory()
File "/var/task/inventory.py", line 123, in call_inventory
inventory[svc] = config.global_inventory[svc]
KeyError: 's3'

how can i pass multiple aws account keys to it and excute the same.

Hey "retribution" almost always has meaning of "punishment". Your readme file has this word but I'm pretty sure you meant "contribution" or "reward" :)

Hello,

Output for sqs queues is always empty whereas other services work fine (s3, ec2, rds..) and obviously there also have sqs queues on the aws account I test.

Running with --log debug does not provide more information on the error but after re-configure the logger manually I can see the following:

Error while processing sqs, eu-west-3, list_queues. Error: argument of type 'NoneType' is not iterable

Has anybody tried this on Google Cloud Shell or in AWS CloudShell?

I use the following commands to start the project:

• git clone https://github.com/janiko71/aws-inventory.git
• cd aws-inventory
• pip3 install -r requirements.txt
• python3 inventory.py

pip3 install gives some errors on AWS Cloudshell, but works fine on GCP.

However, both CloudShells leave me with a non-working application:

I get this error:
Traceback (most recent call last): File "inventory.py", line 61, in <module> ownerId = utils.get_ownerID(profile_name) File "/home/jason_dinkel/729121273778/aws-inventory/res/utils.py", line 173, in get_ownerID identity = sts.get_caller_identity() File "/home/jason_dinkel/.local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/home/jason_dinkel/.local/lib/python3.7/site-packages/botocore/client.py", line 676, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (ExpiredToken) when calling the GetCallerIdentity operation: The security token included in the request is expired

Now, the token comes from the SSO console for GCP and for AWS, the credentials are built-into the cloudshell, so neither should be the problem.

Also, the AWS Cloudshell shouldn't prompt for a [default] profile.
[cloudshell-user@ip-10-0-28-32 aws-inventory]$ python3 inventory.py
Profile name [default] not found, please check.

If anybody wants to have a look, I'd appreciate it. If I find a solution, I will post the steps.

Thanks.

Hi, I ran the inventory script however it took a long time so I exited. I have about ten accounts in my local aws credentials. I would like to specific --profile for one account. Also a secondary thing is to see the output of the running script if possible. Let me know. Thanks!

Hi, is there any reason why kms is in iam.py rather than its own module? elb, elbv2 in networking rather than compute? I'm also trying to add IAM inventory, managed to do users list and groups list, now trying to combine the two - can you think of a way to nest one or the other (e.g. users list under group, or group list under each user?) - 'list-users-for-group' from aws' cli is not really helpful here (no common keys?)

Hi,

I'm planning to use it as a lambda, and save the inventory to a s3 bucket.

S3_INVENTORY_BUCKET looks like a variable to define an output s3 bucket adress, but not used. Is it planned or I missed something ?

Route53 : ends at 95.24% and not 100%. Long execution time.
Hsm: ? Timeout when not existing (in a region).
The script hangs sometimes (throtteling). Account limitation issue?

Thanks for this project. I like the direction its heading. I was able to get it running, it hung for me the first run, but after I started running it for specific services it now completes fine. I'm struggling with the .json output file trying to find a good way to view it. would love suggestions on that. I wish there was an easy way to run it with multiple AWS accounts. even if I could just pass the credentials as command line variables. I could script it to run several different inventories in sequence.

ᐅ python inventory.py s3                                                                        
----------------------------------------------------------------------------------------------------
Number of services   : 1
Services List        : ['s3']
----------------------------------------------------------------------------------------------------

Traceback (most recent call last):! Region : global           ! s3 (list_buckets)                    
  File "inventory.py", line 449, in <module>
    inventory[svc] = config.global_inventory[svc]    
KeyError: 's3'

Note that the above error appears after editing line https://github.com/janiko71/aws-inventory/blob/master/inventory.py#L411 to th.join(60)

Not sure how we can make use of the NextToken parameter to get the full list, but not part of it

$ docker run --name zolotnik-015291475736 -v /tmp/aws-inventory/output:/usr/src/app/output -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID_015291475736} -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY_015291475736} ${REGISTRY_OVH}/zolotnik:latest python ./inventory.py --services s3 ec2 rds dynamodb ecs cloudwatch route53 elb elbv2 ecr

Number of services : 10
Services List : ['s3', 'ec2', 'rds', 'dynamodb', 'ecs', 'cloudwatch', 'route53', 'elb', 'elbv2', 'ecr']

Exception in thread Thread-20:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/src/app/res/awsthread.py", line 48, in run
config.global_inventory[self.aws_service] = self.function_name(*self.arg)
File "/usr/src/app/res/storage.py", line 114, in get_s3_inventory
for objects in page_objects:
File "/usr/local/lib/python3.8/site-packages/botocore/paginate.py", line 269, in iter
response = self._make_request(current_kwargs)
File "/usr/local/lib/python3.8/site-packages/botocore/paginate.py", line 357, in _make_request
return self._method(**current_kwargs)
File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 508, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 915, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (IllegalLocationConstraintException) when calling the ListObjectsV2 operation: The me-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.
Traceback (most recent call last):
File "./inventory.py", line 573, in
inventory[svc] = config.global_inventory[svc]
KeyError: 's3'

Progression is KO when sms and emr in services list for inventory.

(venv) C:\Dev\aws-inventory>py inventory.py --service sns emr
----------------------------------------------------------------------------------------------------
Number of services   : 2
Services List        : ['sns', 'emr']
----------------------------------------------------------------------------------------------------

OwnerID : 559583391002 ! 166.67 % ! Region : us-west-2        ! sns (list_platform_applications)

All inventories are done. Duration: 26.137696 seconds

End of processing.

When Internet connection is unstable, threads wait forever due to the missing timeout in the underline threading's acquire call (I suppose).

Specifying a timeout will ensure the script to ends at some point, writing the information gathered so far.