Script is failing today when attempting to connect to certain regions

`Could not connect to the endpoint URL: "https://datapipeline.us-west-1.amazonaws.com/"

Could not connect to the endpoint URL: "https://datapipeline.ap-southeast-1.amazonaws.com/"

Could not connect to the endpoint URL: "https://datapipeline.sa-east-1.amazonaws.com/"
`

Interestingly, it seems that I am unable to perform an nslookup for the three DNS names above. Other regions seem to resolve properly. Are others experiencing this issue?

Hello!

All the generated CSVs are always empty, except for the "_timestamp__Classic_Platform_Status.csv" that lists this:
us-east-1, Disabled
us-west-1, Disabled
us-west-2, Disabled
eu-west-1, Disabled
ap-southeast-1, Disabled
ap-southeast-2, Disabled
ap-northeast-1, Disabled
sa-east-1, Disabled

Also, the error text file is empty.

I tried with

• python3 py-Classic-Resource-Finder.py -p default
• python3 py-Classic-Resource-Finder.py -p "default"
• python3 py-Classic-Resource-Finder.py -p 'default'
• python3 py-Classic-Resource-Finder.py

But the result is always the same (empty files except for "_Classic_Platform_Status.csv"). But this is strange as in the AWS account, there are at least four "classic load balancers" in eu-west-1.

Any ideas? Thanks!

(The files are generated in a folder with the AWS account id as name. And on the CLI, I can still do "aws ec2 describe-volume-status", "aws s3 ls", "aws ec2 describe-vpc-classic-link", ... . So there's nothing fishy with my cli.)

thanks for this script. good stuff. just wanted to add that by adding a continue statement after line 59 will avoid running unnecessary code so will go alot faster by restarting at the for loop.

as follows:

59 else printf "$region, Disabled\n" >> Classic_Platform_Status.csv ## If supported platforms is only VPC and does not include EC2, output the region and Disabled to a CSV
60 printf "$region, Classic disabled, skipping checks.\n"
61 continue

we use multiple profiles, and need profile support

Hi,

In the Outputs section of the README for the Classic_Platform_Status.csv file may need to also include the "Unknown" status with a recall to check the errors.txt file.
Or/And when the script end and errors.txt file exist write a warning, if you run the script you only see green mark even if all fail which give feeling that all is ok.

Hope it help.

China AWS regions are not supported by Python and Shell scripts.
When I added specific to China regions - it not worked also, probably API should be updated as well. Not sure.

declare -a regions=('cn-north-1' 'cn-northwest-1')

    classicregions = ('cn-north-1', 'cn-northwest-1',)

Expected behavior

Lists contain Classic resources.

Current output

All lists are empty.

error.log file

An error occurred (UnsupportedOperation) when calling the DescribeVpcClassicLink operation: The functionality you requested is not available in this region.

Could not connect to the endpoint URL: "https://datapipeline.cn-north-1.amazonaws.com.cn/"

Could not connect to the endpoint URL: "https://opsworks.cn-north-1.amazonaws.com.cn/"

An error occurred (UnsupportedOperation) when calling the DescribeVpcClassicLink operation: The functionality you requested is not available in this region.

Could not connect to the endpoint URL: "https://datapipeline.cn-northwest-1.amazonaws.com.cn/"

Could not connect to the endpoint URL: "https://opsworks.cn-northwest-1.amazonaws.com.cn/"

Readme says I need to have these permissions:

elasticloadbalancing:DescribeLoadBalancers
elasticmapreduce:DescribeCluster
elasticmapreduce:ListBootstrapActions
elasticmapreduce:ListClusters
elasticmapreduce:ListInstanceGroups
opsworks:DescribeStacks

Editing my policy on the IAM Mangement Console, there doesn't seem to any service for elasticloadbalancing and elasticmapreduce (unless they're named completely differently than these keys).

In addition, when I try to add opsworks:DescribeStacks, upon saving it's not set.

When I run the tool, I get a few errors saying I'm missing:

• elasticloadbalancing:DescribeLoadBalancers
• elasticmapreduce:ListClusters
• opsworks:DescribeStacks

Not sure if this indicates these things are already retired, or I'm not using them or what.

I ran the script, and it picked out one region to give something unusual.

Most of the CSVs are empty, except for Classic_Platform_Status.csv and Classic_SGs.csv ; so I found that Classic_SGs.csv contains a solitary security group named "default" in us-east-1, whereas I have no other resources in that region.

I saw that there is a closed issue (#18 ) where the person requested for more info on how to deal with such reports of a default security group being included in Classic_SGs.csv for no apparent reason.

Could the script be fixed instead to not include these default security groups in Classic_SGs.csv ? I don't understand why it doesn't include the default security groups in other regions, but only this particular region?

I also have a related stackoverflow question about this: https://stackoverflow.com/questions/69787759/ec2-classic-resource-that-cant-be-deleted-now-what?noredirect=1#comment123433839_69787759

My AWS config requires MFA to assume a role.

I can understand if this tool prompted for MFA each time it assumed a role, once for each account (even though that is not necessary if implemented right).

However, this tool prompts once for every region in every account. This makes the tool unusable for me.

The resource finder does not notify you when your ElasticBeanstalk Environments contain an active classic loadbalancer only if they aren't in a VPC. Since all CLBs are going away per Amazon direct support inquiry, the script should be updated to notice that your environment it utilizing one.

The Classic LBs also do not show up in Classic_CLBs.csv.

Hi,

There's some missing information on how to deal with default security groups.
After several run and cleanup all the csv files were empty (need to remove the files between each run as the script append data) except Classic_SGs.csv which list several SG in different region but all are default and when trying to delete in the interface this message is displayed:

This is a default security group. Default security groups cannot be deleted

If you can add some information on how to deal with this and if it's ok to request a conversion to VPC-Only for this case.

Other small improvement that may help when run the script several time, if you can append a timestamp for each line in the logs it can be helpful to see difference between the several run.

#!/usr/bin/python3 isn't good practice
Use #!/usr/bin/env python3 instead

The way it's currently written screws things up if you try and use a venv or asdf.

We just started to run this script on our accounts/environments and we are noticing a 'flapping' where the tool flags issues in ElasticBeanstalk Environments not having a VPC and in each case the Environment has recently been terminated. E.g., of course it doesn't have a VPC associated because the resources have been removed.

Looking at this line I wonder if it should be changed to only consider Environments that are Ready?

Hello!

The end output of the script shows: "If no resources were found in EC2-Classic for a service, there was no CSV created."

However, each time this has been run, it has produced empty CSV files for each resource it checked other than the status CSV. When looking at the Classic_Platform_Status.csv, it shows that ec2_classic is disabled in all regions.

(the script was run with creds that have the correct access)

Is the line that is output at the end of the script correct and I should expect no CSVs if nothing is found? If not, it should be corrected and also reflected in the README.

Thanks!

First time running this locally - it's a great tool, thanks for writing it! I'm using the latest version.

I encountered this issue on my first try:

$ python3 py-Classic-Resource-Finder.py --profile production
Checking the Classic platform status in us-east-1
Checking for EIPs in us-east-1
Checking for Classic EC2 Instances in us-east-1
Checking for Classic Security Groups in us-east-1
Checking for VPCs with ClassicLink enabled in us-east-1
Checking for AutoScaling Groups configured for Classic in us-east-1
Checking for Classic Load Balancers running in EC2-Classic in us-east-1
Checking for Classic RDS Instances in us-east-1
Checking for Classic ElastiCache clusters in us-east-1
Checking for Classic Redshift clusters in us-east-1
Checking for Classic Elastic BeanStalk Environments in us-east-1
Checking for Classic EMR clusters in us-east-1
Checking for Classic OpsWorks stacks in us-east-1
Checking for Classic Data Pipelines in us-east-1
Traceback (most recent call last):
  File "/Users/scott/src/external/ec2-classic-resource-finder/py-Classic-Resource-Finder.py", line 847, in <module>
    main(argparser(sys.argv[1:]))
  File "/Users/scott/src/external/ec2-classic-resource-finder/py-Classic-Resource-Finder.py", line 841, in main
    loopregions(classicregions, datapipelineregions, creddict)
  File "/Users/scott/src/external/ec2-classic-resource-finder/py-Classic-Resource-Finder.py", line 668, in loopregions
    process.start()
  File "/usr/local/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 115, in start
    assert self._popen is None, 'cannot start a process twice'
AssertionError: cannot start a process twice

As you can see above, I'm running Python 3.9.9:

$ python3 --version
Python 3.9.9

I did get partial results from the single region and the errors file was empty after switching to my mfa profile.

Any hints?

Version: ea774c5

We got a Beanstalk application with a space in it's name (Public Page), so L220 fails to describe the configuration.

Result:
In errors.txt you'll get:

An error occurred (InvalidParameterValue) when calling the DescribeConfigurationSettings operation: No Environment found for EnvironmentName = 'Page'.

and in Classic_ElasticBeanstalk_Applications_Environments.csv:

Public, Page, eu-west-1

Currently the tool assumes all known AWS Regions. Which is great as it makes the validate analysis complete!

However, if my organization has limited ourselves to specific regions the tool does not take that into consideration. It wastes a TON of time looking at regions that are not interesting.

Provide a means where the user can provide an argument to limit the regions that are validated.

PS. This issue/suggestion would probably never occurred if some form of proc forks were used. In that the main regions we are using take a lot of time to process and the unused could have easily just been done in the background and never resulted in any useful information.

Example App name: "Webserver App"
Example Env Name: "webserver-app-a"

In the script, currently, the delimiter being used for cut is ' ' (space)
for ebenvapp in jq -r '.Environments[] | .ApplicationName +" "+ .EnvironmentName' <<< $ebappraw 2>> errors.txt ## Loop over each application and environment pair
do
ebapp=cut -d " " -f1 <<< $ebenvapp 2>> errors.txt ## Extract the Application name
ebenv=cut -d " " -f2 <<< $ebenvapp 2>> errors.txt ## Extract the Environment name

• this results in ebapp: "Webserver" & ebenv: "App"

Changing this to delimiter of ':' (colon) is a possible fix tested in our environment with spaces.
for ebenvapp in jq -r '.Environments[] | .ApplicationName +":"+ .EnvironmentName' <<< $ebappraw 2>> errors.txt ## Loop over each application and environment pair
do
ebapp=cut -d ':' -f1 <<< $ebenvapp 2>> errors.txt ## Extract the Application name
ebenv=cut -d ':' -f2 <<< $ebenvapp 2>> errors.txt ## Extract the Environment name

• this results in ebapp: "Webserver App" & ebenv: "webserver-app-a"

-Not sure if this is necessary, but in the next line I also put double quotes ("") around the $ebapp
ebnsval=aws elasticbeanstalk describe-configuration-settings --application-name **"$ebapp"** --environment-name $ebenv --query 'ConfigurationSettings[*].OptionSettings[?Namespace==\aws:ec2:vpc`&&OptionName==`VPCId`&&Value!=`null`].OptionName' --region $region --output text 2>> errors.txt` ## If the environment is configured for a vpc return "VPCId"

The script currently checks if there are any VPC security groups attached to an RDS instance to determine if it is a classic instance or not. I believe the API call to check this must occasionally fail or be throttled and that the script does not handle this properly creating a false positive. Each time I ran the script across my organization of several hundred accounts and RDS instances I got a handful of false positives which were different each time.