You say "The simplest regular expression is " which is wrong, since "" is not a valid regex. The examples in the README following that then go on to mix regex and glob arbitrarily, which is quite confusing.

Hi,

I'm curious if this is still being worked on or being abandoned ? If it's being abandoned, could the code be transferred to someone who is willing to work on it? This is incredibly useful, but could be extended to be even more powerful..

Thanks!

Even I'm sure the resource has tags. Any suggestion? Thanks.

uri = '::sqs:::/'
for resource in skew.scan(uri):
print(resource.tags)

Cloudfront resources are global, but they're being scanned for each region:

Scanning arn:aws:cloudfront::123:*/*
arn:aws:cloudfront::123:distribution/ABC
arn:aws:cloudfront::123:distribution/DEF
arn:aws:cloudfront::123:distribution/ABC
arn:aws:cloudfront::123:distribution/DEF
arn:aws:cloudfront::123:distribution/ABC
arn:aws:cloudfront::123:distribution/DEF
...

Producing attribute type error. Config is being cast incorrectly. Was wondering if you could possibly shed some light?

All configurations files have been set properly. I have done interpreter calls to each function and produced the desired dictionary that should reside in config. However the call directed below will error out.

Below is the call and error.

from skew.arn import ARN
ARN()

AttributeError Traceback (most recent call last)
in ()
----> 1 ARN()

/usr/local/lib/python2.7/dist-packages/skew/arn/init.pyc in init(self, arn_string)
257 self.query = None
258 self._components = None
--> 259 self._build_components_from_string(arn_string)
260
261 def repr(self):

/usr/local/lib/python2.7/dist-packages/skew/arn/init.pyc in _build_components_from_string(self, arn_string)
291 pairs = zip_longest(
292 self.ComponentClasses, arn_string.split(':', 6), fillvalue='*')
--> 293 self._components = [c(n, self) for c, n in pairs]
294
295 @Property

/usr/local/lib/python2.7/dist-packages/skew/arn/init.pyc in init(self, pattern, arn)
143
144 def init(self, pattern, arn):
--> 145 self._account_map = self._build_account_map()
146 super(Account, self).init(pattern, arn)
147

/usr/local/lib/python2.7/dist-packages/skew/arn/init.pyc in _build_account_map(self)
157 session.profile = profile
158 config = session.get_scoped_config()
--> 159 account_id = config.get('account_id')
160 if account_id:
161 account_map[account_id] = profile

AttributeError: 'str' object has no attribute 'get'

For example, a Resource Record Set in Route53 is subordinate to the Hosted Zone within which it is contained. There may be other examples. There is currently no way to represent this in skew.

Fresh install w/ Vagrant xenial64 (also on osx). Using pip install for release.

When trying to scan IAM or route53 (aka no region services), code just returns.

ubuntu@ubuntu-xenial:~/.local/lib/python2.7/site-packages/skew$ find . -name "*.pyc" -print
./exception.pyc
./config.pyc
./resources/__init__.pyc
./__init__.pyc
./arn/__init__.pyc

sample code:

import skew
arn = skew.scan('arn:aws:route53:us-west-2:xxxxxxxxxxxxxx:*/*')                                                                          
for resource in arn:
    print(resource)
    print(resource.data)

else it can cause skew installation errors when used as a dependency due to mismatched pypi versions across transitive dep set on pyyaml otherwise. ie. if one dep depends on any version greater than pyyaml > 3.11 and skew in current form pins it to 3.11 and 3.12 gets installed for the other dep, then skew install barfs.

simple fix was already merged in #97 but needs a release to close the loop.

I downloaded skew using pip and tried the script mentioned here: https://stackoverflow.com/questions/30674938/how-to-progammatically-list-all-aws-resources-and-tags
and I get the following error:

/usr/local/lib/python2.7/dist-packages/skew/resources/resource.pyc in enumerate(cls, arn, region, account, resource_id, aws_creds)
     26     def enumerate(cls, arn, region, account, resource_id=None, aws_creds=None):
     27         client = skew.awsclient.get_awsclient(
---> 28             cls.Meta.service, region, account, aws_creds=aws_creds)
     29         kwargs = {}
     30         do_client_side_filtering = False

TypeError: get_awsclient() got an unexpected keyword argument 'aws_creds'

My .skew file looks like this:

accounts:
  "<ACCOUNTID>":
    profile: default

(with replaced with the actual ID).

I also tried altering my aws-cli config to change the default profile to a named one but got the same error.

see https://twitter.com/ywxwy/status/524042098761555969

Connecting to any S3 endpoint and listing the buckets will give you all buckets owned by the account. Enumerating a specific region should only return a bucket whose location constraint is set to that region.

Thank you so much for not only creating this package and all of your efforts that have enriched the Cloud Computing Universe. I have lambda functions that gather information about our AWS organization. We use assume_role to get access_keys and secret access keys with session tokens. I'm struggling to develop a method to implement the skew config requirements for this package into our lambda functions. Currently we access account lists from a rds repository (not all accounts are in the aws organization) and build role arns and role sessions in a lambda "manager" function. We then create an inline policy and attach it to the lambda worker execution function role before invoking (using an function alias arn for our dev, test, and prod stages) our lambda worker function that gathers the information we need. We would prefer not to use Lambda environment variables and any other methods that involve reading a profile from an external source for concerns about security and maintenance in the function's lifecycle. Is there any recommendation or can you point me in direction where someone has addressed this issue so that I can use the sts response of access keys, secret access keys instead of the skew config file. Thank you. jm

I have a following simple code:
`
#!/usr/bin/python

import skew

arn = skew.scan('arn:aws::::/*')
for resource in arn:
print(resource)
`
When I run it I get following error:

arn:aws:apigateway:us-east-1:725326807264:restapis/zmgw9h76u7
Traceback (most recent call last):
File "./scan_failing.py", line 6, in
for resource in arn:
File "/usr/local/lib/python2.7/dist-packages/skew/arn/init.py", line 319, in iter
for scheme in self.scheme.enumerate(context, **self.kwargs):
File "/usr/local/lib/python2.7/dist-packages/skew/arn/init.py", line 244, in enumerate
context, **kwargs):
File "/usr/local/lib/python2.7/dist-packages/skew/arn/init.py", line 229, in enumerate
context, **kwargs):
File "/usr/local/lib/python2.7/dist-packages/skew/arn/init.py", line 214, in enumerate
context, **kwargs):
File "/usr/local/lib/python2.7/dist-packages/skew/arn/init.py", line 195, in enumerate
context, **kwargs):
File "/usr/local/lib/python2.7/dist-packages/skew/arn/init.py", line 146, in enumerate
context, **kwargs):
File "/usr/local/lib/python2.7/dist-packages/skew/arn/init.py", line 128, in enumerate
self._arn, region, account, resource_id, **kwargs))
File "/usr/local/lib/python2.7/dist-packages/skew/resources/resource.py", line 28, in enumerate
cls.Meta.service, region, account, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/skew/awsclient.py", line 139, in get_awsclient
return AWSClient(service_name, region_name, account_id, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/skew/awsclient.py", line 51, in init
self._client = self._create_client()
File "/usr/local/lib/python2.7/dist-packages/skew/awsclient.py", line 81, in _create_client
return session.client(self.service_name, region_name=self.region_name)
File "/usr/local/lib/python2.7/dist-packages/boto3/session.py", line 250, in client
aws_session_token=aws_session_token, config=config)
File "/usr/local/lib/python2.7/dist-packages/botocore/session.py", line 809, in create_client
client_config=config, api_version=api_version)
File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 62, in create_client
verify, credentials, scoped_config, client_config)
File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 188, in _get_client_args
timeout=(new_config.connect_timeout, new_config.read_timeout))
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 240, in create_endpoint
raise ValueError("Invalid endpoint: %s" % endpoint_url)
ValueError: Invalid endpoint: https://iam..amazonaws.com

I am not sure whether it is a bug, or it is illegal pattern. Please can somebody advice?

I would be like to be able to query for all ELB's, and also retrieve their Attributes, and Policies.

Ok I'll be the first to log a ticket.

I've assumed that generally your library can be made more performant by adding multithreading.

For some calls, like grabbing a list of all autoscaling groups, a single call pulls back all the data for a single region. For this reason, to start with maybe the connections to the regions could be multithreaded with as many threads as there are regions.

This allows all the network calls to happen concurrently across regions.

Should speed things up markedly if the library doesn't already do this.

Of course, this adds complexity but it should be worth it in the speed gains.

Cheers.

Summary

So I've been playing with skew for a week or so now and whilst it principally works for what I want it to do at a functional level, I've found that something is causing it to handle blocking IO very poorly (not at all in fact)

The tests I've been running are to scan my own AWS account for S3 buckets. Skew works perfectly for finding the 3 buckets I expected it to, but then never returns.

At first I thought I was being impatient, so I decided to leave it run overnight and through the morning. The function still never returned. I understand that scanning all of AWS is a potentially non-trivial task, but more than 16 hours? Something must be up.

Investigation

The first thing I noticed trying to work out what was going on here was (when it worked) the dump from the keyboard interrupt when running in the console:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ares/.local/lib/python3.8/site-packages/skew/arn/__init__.py", line 323, in __iter__
    for scheme in self.scheme.enumerate(context, **self.kwargs):
  File "/home/ares/.local/lib/python3.8/site-packages/skew/arn/__init__.py", line 247, in enumerate
    for provider in self._arn.provider.enumerate(
  File "/home/ares/.local/lib/python3.8/site-packages/skew/arn/__init__.py", line 232, in enumerate
    for service in self._arn.service.enumerate(
  File "/home/ares/.local/lib/python3.8/site-packages/skew/arn/__init__.py", line 217, in enumerate
    for region in self._arn.region.enumerate(
  File "/home/ares/.local/lib/python3.8/site-packages/skew/arn/__init__.py", line 198, in enumerate
    for account in self._arn.account.enumerate(
  File "/home/ares/.local/lib/python3.8/site-packages/skew/arn/__init__.py", line 145, in enumerate
    for resource in self._arn.resource.enumerate(
  File "/home/ares/.local/lib/python3.8/site-packages/skew/arn/__init__.py", line 127, in enumerate
    resources.extend(resource_cls.enumerate(
  File "/home/ares/.local/lib/python3.8/site-packages/skew/resources/aws/s3.py", line 27, in enumerate
    resources = super(Bucket, cls).enumerate(arn, region, account,
  File "/home/ares/.local/lib/python3.8/site-packages/skew/resources/resource.py", line 54, in enumerate
    data = client.call(enum_op, query=path, **kwargs)
  File "/home/ares/.local/lib/python3.8/site-packages/skew/awsclient.py", line 127, in call
    data = op(**kwargs)
  File "/home/ares/.local/lib/python3.8/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ares/.local/lib/python3.8/site-packages/botocore/client.py", line 691, in _make_api_call
    http, parsed_response = self._make_request(
  File "/home/ares/.local/lib/python3.8/site-packages/botocore/client.py", line 711, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/ares/.local/lib/python3.8/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/ares/.local/lib/python3.8/site-packages/botocore/endpoint.py", line 134, in _send_request    success_response, exception = self._get_response(
  File "/home/ares/.local/lib/python3.8/site-packages/botocore/endpoint.py", line 166, in _get_response    success_response, exception = self._do_get_response(
  File "/home/ares/.local/lib/python3.8/site-packages/botocore/endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "/home/ares/.local/lib/python3.8/site-packages/botocore/endpoint.py", line 269, in _send
    return self.http_session.send(request)
  File "/home/ares/.local/lib/python3.8/site-packages/botocore/httpsession.py", line 344, in send
    urllib_response = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 421, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1344, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
KeyboardInterrupt
>>>

Diving into the codebase skew wraps around boto3 which in turn uses (to no massive surprise) urllib3 to manage its requests to AWS itself.

So cool, it's just urllib3 under the hood and it's timing out. Well urllib3 doesn't have a default timeout for requests, but you can certainly set one. So using socket.setdefaulttimeout(60) set the timeout to 60 seconds (pretty fair imo) and suddenly skew (or more specifically boto3) doesn't work at all. Even the original buckets that I expect to see listed there are not returned. What makes this slightly more confusing is that it doesn't actually return a timeout exception; I'm not even convinced the requests are made.

With messing with the timeout not working I thought, what if I can just force the thread to exit by force? So using various forms of inspiration from this thread, I tried forcing skew to exit early.

Context Manager

This context manager didn't work at all, skew just didn't stop

@contextmanager
def timeout(duration):
    def timeout_handler(signum, frame):
        raise BlockingIOError(f'Function timed out after {duration} seconds')

    signal.signal(signal.SIGALRM, timeout_handler)
    signal.alarm(duration)
    yield
    signal.alarm(0)

Thread Decorator

This thread decorator saw better results in that the quit_function call is actually made (the call to log.info is made) but again, skew doesn't exit gracefully and just hangs.

def quit_function(fn_name):
    log.info(f'{fn_name} took too long')
    thread.interrupt_main()  # raises KeyboardInterrupt


def exit_after(s):
    """
    use as decorator to exit process if
    function takes longer than s seconds
    """

    def outer(fn):
        def inner(*args, **kwargs):
            timer = threading.Timer(s, quit_function, args=[fn.__name__])
            timer.start()
            try:
                result = fn(*args, **kwargs)
            finally:
                timer.cancel()
            return result

        return inner

    return outer

I'm honestly running out of ideas as to where to go looking for this one. My use case is actually to be able to run skew on a serverless function endpoint that I can just call, but these have well defined timeouts that I just can't seem to get skew to obey.

Any ideas?

Hi folks, this thing is a life saver on the pubcloud side.
But not working with govcloud is a shame.
Any thoughts on adding in the govcloud region?

thanks

steve

After your latest update I no longer get the aws_creds error when running this script: http://bit.ly/1JJ0TH2
Thankyou. However I now get another error:

/usr/local/lib/python2.7/dist-packages/botocore/client.pyc in _make_api_call(self, operation_name, api_params)
    405 
    406         if http.status_code >= 300:
--> 407             raise ClientError(parsed_response, operation_name)
    408         else:
    409             return parsed_response

ClientError: An error occurred (AccessDenied) when calling the DescribeStacks operation: User: arn:aws:iam::<ID>:user/<NAME> is not authorized to perform: cloudforma
tion:DescribeStacks

import skew
arn = skew.scan('arn:aws:ec2:::instance/')
for instance in arn:
... print(arn)
...
arn:aws:ec2:::instance/
arn:aws:ec2:::instance/*
arn:aws:ec2:::instance/*
arn:aws:ec2:::instance/*
arn:aws:ec2:::instance/*
arn:aws:ec2:::instance/*
arn:aws:ec2:::instance/*
arn:aws:ec2:::instance/*

When I use resource.data and explicitly choose an individual ARN it works fine and I get the results -

from skew import scan
arn = skew.scan('arn:aws:ec2:eu-west-1:093228123150:instance/i-0fdffef58576f8027')
Traceback (most recent call last):
File "", line 1, in
NameError: name 'skew' is not defined
arn = scan('arn:aws:ec2:eu-west-1:XXXXXXXXXXX:instance/i-0fdfXXXXXXXX027')
for resource in arn:
... print(resource.data)
...
{u'Monitoring': {u'State': 'disabled'}, u'PublicDnsName': 'ec2-52-XX-139-XX.eu-west-1.compute.amazonaws.com', u'State': {u'Code': 16, u'Name': 'running'}, u'EbsOptimized': True, yada yada yada

My end goal is to pull ARNs for all AWS resources within an AWS Account. Trying to get the script at the bottom of the stackoverflow (http://stackoverflow.com/questions/30674938/how-to-progammatically-list-all-aws-resources-and-tags) working but that's not either.

Any help gratefully received!

This is a fantastic utility, I upgraded to skew==0.10.0 this morning (botocore==0.81.0) and noticed I'm not getting back some of the total resources.

For instance using the untag example it returns 306 Total Instances across both our accounts, we have 428 total across the two accounts, it is in fact returning some of the instances from both accounts.

Same with ELB queries - Skew returns 98 we have 181.

Here are the arn's I'm using, we are all in us-east-1 for now:
arn:aws:ec2:us-east-1::instance/
arn:aws:elb:us-east-1::loadbalancer/

Is there a limit to number of resources returned now that I'm missing either in skew or botocore?

Thanks, for now I'll try 0.8.0 and see if it gets me back to all resources.

Within https://github.com/scopely-devops/skew/blob/develop/skew/arn/__init__.py#L168 there is a missing comma, this results in the region name for af-south-1 and ca-central-1 being merged together to form af-south-1ca-central-1

The skewer catch all query arn:aws:*:*:*:*/*, or even a variation with a specific account, say arn:aws:*:*:123456789012:*/* always includes all publicly 'owned' images/snapshots, because this is the default API behavior of DescribeImages and DescribeSnapshots in absence of any filter, rather than the more sensible (I think) inverse behavior of only listing the self owned ones and requiring an opt in parameter to get all.

I realize the dilemma regarding API coverage here, but with respect to the use case, I'd assume that even more than the typical 80/20 fraction of users might be interested in 'self' owned images/snapshots only and would prefer skew to restrict the results to this (opinionated) choice accordingly?!

I would like to see support for retrieving RDS Parameter Groups. I'll open a Pull Request for this soon.

In version 0.16.1, when running a skew.scan() for a wildcard account and or region the following error is encountered:

raise ClientError(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidInstanceID.NotFound) when calling the DescribeInstances operation: The instance ID <SNIP INSTANCE ID>' does not exist

The call for this looks like:
skew.scan('arn:aws:ec2:*:*:instance/<SPECIFIC INSTANCE ID>')

If I call this with the specific region and instance id, then the query succeeds.

The full stack trace after sniping out my code lines:

#!/usr/bin/env python
(...)
  File "/usr/local/lib/python2.7/dist-packages/skew/arn/__init__.py", line 319, in __iter__
    for scheme in self.scheme.enumerate(context, **self.kwargs):
  File "/usr/local/lib/python2.7/dist-packages/skew/arn/__init__.py", line 244, in enumerate
    context, **kwargs):
  File "/usr/local/lib/python2.7/dist-packages/skew/arn/__init__.py", line 229, in enumerate
    context, **kwargs):
  File "/usr/local/lib/python2.7/dist-packages/skew/arn/__init__.py", line 214, in enumerate
    context, **kwargs):
  File "/usr/local/lib/python2.7/dist-packages/skew/arn/__init__.py", line 195, in enumerate
    context, **kwargs):
  File "/usr/local/lib/python2.7/dist-packages/skew/arn/__init__.py", line 146, in enumerate
    context, **kwargs):
  File "/usr/local/lib/python2.7/dist-packages/skew/arn/__init__.py", line 128, in enumerate
    self._arn, region, account, resource_id, **kwargs))
  File "/usr/local/lib/python2.7/dist-packages/skew/resources/resource.py", line 49, in enumerate
    data = client.call(enum_op, query=path, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/skew/awsclient.py", line 116, in call
    data = results.build_full_result()
  File "/usr/local/lib/python2.7/dist-packages/botocore/paginate.py", line 271, in build_full_result
    for response in self:
  File "/usr/local/lib/python2.7/dist-packages/botocore/paginate.py", line 85, in __iter__
    response = self._make_request(current_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/paginate.py", line 157, in _make_request
    return self._method(**current_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 310, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 407, in _make_api_call
    raise ClientError(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidInstanceID.NotFound) when calling the DescribeInstances operation: The instance ID ' <SNIP INSTANCE ID>' does not exist

Hi,

It seems like AWS CloudSearch domains aren't found by a scan operation. It doesn't matter if I provide an ARN pattern or a specific ARN. For a lot of other AWS resources it works fine.

$ aws cloudsearch --region eu-central-1 list-domain-names
{
    "DomainNames": {
        "dev-cloudsearch": "2013-01-01"
    }
}

$ aws cloudsearch --region eu-central-1 describe-domains | jq .DomainStatusList[0].ARN
"arn:aws:cloudsearch:eu-central-1:182674520467:domain/dev-cloudsearch"

#!/usr/bin/python

import sys
from skew import scan

for arg in sys.argv:
	arn = scan(arg)
    	for resource in arn:
        	print(resource)

• nothing: $ ./list-by-arn.py arn:aws:cloudsearch:::domain/*
• nothing: $ ./list-by-arn.py arn:aws:cloudsearch:eu-central-1:182674520467:domain/dev-cloudsearch
• works fine: $ ./list-by-arn.py arn:aws:ec2:::instance/*
• works fine: $ ./list-by-arn.py arn:aws:s3:::bucket/*

Is CloudSearch not supported?

caniusepython3 reports that skew won't work for me as we move to python3, which makes me very sad.

bash-3.2$ caniusepython3 --requirement ../requirements.txt 
Finding and checking dependencies ...

You need 1 project to transition to Python 3.
Of that 1 project, 1 has no direct dependencies blocking its transition:

  skew

FYI, this is the requirements.txt for that example.

bash-3.2$ cat ../requirements.txt 
PyYAML
future
futures
arrow
boto
boto3
functools32
lazy
munch
netaddr
plumbum
pycrypto
pymongo
skew
slackclient
tabulate
troposphere
networkx
docopt

If you have an ARN like:

arn:aws:*:*:*:*/*

and you try to enumerate it, you get the error:

KeyError: 'aws.autoscaling.*'"

Substituting the wildcard with a specific service works correctly.

When I install skew using pip 1.3.1 from /usr/lib/python2.6/site-packages (python 2.6), running Python 2.6.9 (tested on Amazon Linux), the pip install succeeds with the following message at the end:

Successfully installed skew botocore jmespath simplejson

But when I try to import skew, I get the following error:

$ python
Python 2.6.9 (unknown, Sep 13 2014, 00:25:11)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import skew
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/site-packages/skew/__init__.py", line 19, in <module>
    from skew.arn import ARN
  File "/usr/lib/python2.6/site-packages/skew/arn/__init__.py", line 22, in <module>
    import skew.resources
  File "/usr/lib/python2.6/site-packages/skew/resources/__init__.py", line 14, in <module>
    import importlib
ImportError: No module named importlib
>>>

This gets fixed if I do a pip install importlib.

I would want importlib to get installed as a dependency when I do a pip install skew.

Hi,
currently the ARN pattern returned from an elasticbeanstalk environment does not include the application name.
See Beanstalk ARN pattern reference

Expected
arn:aws:elasticbeanstalk:REGION:ACCOUNT-ID:environment/APPLICATION-NAME/ENVIRONMENT-NAME

Actual
arn:aws:elasticbeanstalk:REGION:ACCOUNT-ID:environment/ENVIRONMENT-NAME

I think with some hints I can provide a fix for this issue. Unfortunately I am a bit unsure where the additional field application name best fits into the current implementation.
The call to describe_environments returns the application name field but I am not sure how to access it in the environment class!

In version 0.16.1 I'm running a query against 3 accounts and defining the accounts in the code using:

  skew.config._config = {
      'accounts': {
          'ACCT1' : {'profile': None},
          'ACCT2' : {'profile': None},
          'ACCT3' : {'profile': None},
       }
  }

When I run a scan for an ec2 instance id, I get back positive results from all 3 accounts even though the instance only exists in 1 account.

While investigating cloud-custodian/cloud-custodian#541, I think I may have found a typo.

Shouldn’t the filter be set to “GroupIds” on this line?

https://github.com/scopely-devops/skew/blob/develop/skew/resources/aws/ec2.py#L45

I'd like to be able to retrieve IAM Policy Documents. The aws command requires a specific "policy name" and "policy doc version." Would I be able to code this, by setting the "detail_spec" property and writing a custom iter method, similar to the CloudFormation class?

Since some recentish updates to boto3/botocore skew no longer works for iam/route53

It will throw value errors as follows:

ValueError: Invalid endpoint: https://iam..amazonaws.com

Note the attempts at a blank region. Updates to boto3/botocore expect the region_name to be passed to the client as None if there is no region as opposed to empty string.

This appears to have cropped up due to the way the version is pinned in requirements.txt as a boto3 >=
Since this is a breaking change that came for upstream and there hasn't been much activity recently the breaking change has gone unnoticed.

A fix for this specific issue can be found in #95

You may also wish to consider pinning boto3/botocore to specific versions and use something like https://requires.io to auto make prs when dependencies version's creep. This should allow you to be alerted on failed tests based on upstream changes while keeping your code in a working state when installing from pypi.

I tested a skew script in god_mode and it worked perfectly as it should. However, it would be great to know the permissions required so I can create an IAM role for the script. Instead you could also mention the kind of operations skew performs and I can make a policy accordingly. Thank you for the help.

This is an awesome module! Fills a real need. Thank you so much!

I assume this is odd...

For example here it's scanning for an instance with a fully specified ARN

In [44]: for i in scan('arn:aws:ec2:us-west-2:%s:instance/%s' % (my_acnt, mystery_instance)): print i.data['InstanceType'], i.data['LaunchTime']
c3.large 2017-01-10 21:39:11+00:00

But if I wild card the region i get nothing.

In [45]: for i in scan('arn:aws:*:%s:instance/%s' % (my_acnt, mystery_instance)): print i.data['InstanceType'], i.data['LaunchTime']

Dis is the version pip gave me.

In [46]: skew.__version__
Out[47]: '0.16.1'

Looks like an issue only when querying against multiple accounts.
This was my scan:
arn = scan('arn:aws:ec2:us-east-1::instance/')

This is part of the result that was concerning to me:
ARN - arn:aws:ec2:us-east-1:ACCOUNT1:instance/i-534cac01
ARN - arn:aws:ec2:us-east-1:ACCOUNT2:instance/i-534cac01

I tried to work around it for now by querying both accounts separately and I'm only getting results from ACCOUNT1. I'll keep digging but hoping this is something others have seen as well. I'm on version 0.10.1

In [3]: from skew import scan

In [4]: arn = scan('arn:aws:ec2:us-west-2::instance/')

In [5]: arn
Out[5]: arn:aws:ec2:us-west-2::instance/

am I missing something? My ~/.skew and ~/.aws/credentials files are setup. I assume it would crawl the account I have listed in ~/.skew. Though I don't see any output or anything. I can't iterate over arn.

Using this sample code:

import skew
from skew import scan
from pprint import pprint
arn = scan('arn:aws:*:*:*:*/*')
for resource in arn:
    print(resource.data)

Has error of:

Traceback (most recent call last):
  File "/Users/BKruger/skewer.py", line 7, in 
    for resource in arn:
  File "/Users/BKruger/Library/Python/2.7/lib/python/site-packages/skew/arn/__init__.py", line 319, in __iter__
    for scheme in self.scheme.enumerate(context, **self.kwargs):
  File "/Users/BKruger/Library/Python/2.7/lib/python/site-packages/skew/arn/__init__.py", line 244, in enumerate
    context, **kwargs):
  File "/Users/BKruger/Library/Python/2.7/lib/python/site-packages/skew/arn/__init__.py", line 229, in enumerate
    context, **kwargs):
  File "/Users/BKruger/Library/Python/2.7/lib/python/site-packages/skew/arn/__init__.py", line 214, in enumerate
    context, **kwargs):
  File "/Users/BKruger/Library/Python/2.7/lib/python/site-packages/skew/arn/__init__.py", line 195, in enumerate
    context, **kwargs):
  File "/Users/BKruger/Library/Python/2.7/lib/python/site-packages/skew/arn/__init__.py", line 146, in enumerate
    context, **kwargs):
  File "/Users/BKruger/Library/Python/2.7/lib/python/site-packages/skew/arn/__init__.py", line 128, in enumerate
    self._arn, region, account, resource_id, **kwargs))
  File "/Users/BKruger/Library/Python/2.7/lib/python/site-packages/skew/resources/resource.py", line 28, in enumerate
    cls.Meta.service, region, account, **kwargs)
  File "/Users/BKruger/Library/Python/2.7/lib/python/site-packages/skew/awsclient.py", line 139, in get_awsclient
    return AWSClient(service_name, region_name, account_id, **kwargs)
  File "/Users/BKruger/Library/Python/2.7/lib/python/site-packages/skew/awsclient.py", line 51, in __init__
    self._client = self._create_client()
  File "/Users/BKruger/Library/Python/2.7/lib/python/site-packages/skew/awsclient.py", line 81, in _create_client
    return session.client(self.service_name, region_name=self.region_name)
  File "/Users/BKruger/Library/Python/2.7/lib/python/site-packages/boto3/session.py", line 256, in client
    aws_session_token=aws_session_token, config=config)
  File "/Users/BKruger/Library/Python/2.7/lib/python/site-packages/botocore/session.py", line 824, in create_client
    client_config=config, api_version=api_version)
  File "/Users/BKruger/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 68, in create_client
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/Users/BKruger/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 130, in _get_client_args
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/Users/BKruger/Library/Python/2.7/lib/python/site-packages/botocore/args.py", line 72, in get_client_args
    timeout=(new_config.connect_timeout, new_config.read_timeout))
  File "/Users/BKruger/Library/Python/2.7/lib/python/site-packages/botocore/endpoint.py", line 240, in create_endpoint
    raise ValueError("Invalid endpoint: %s" % endpoint_url)
ValueError: Invalid endpoint: https://iam..amazonaws.com

The problem is in:
skew/arn/init.py, line 169 - there's an entry made of '' that shouldn't be, it should just be an empty list.

old code:
_no_region_required = ['']
fix:
_no_region_required = []

This is just for documenting purposes, a pull request will need to be made yet.

It'd be nice to have a new release containing all the latest changes.

When enumerating the stack resources in Cloudformation, it is not guaranteed that each resource will have a PhysicalResourceId. This is causing a KeyError when enumerating such a resource.