twisted / txaws Goto Github PK

View Code? Open in Web Editor NEW

32.0 32.0 18.0 4.16 MB

Twisted-based Asynchronous Libraries for Amazon Web Services and clouds that support the AWS APIs

License: MIT License

Python 99.47% Makefile 0.50% Shell 0.01% Gherkin 0.03%

txaws's Introduction

Twisted

For information on changes in this release, see the NEWS file.

What is this?

Twisted is an event-based framework for internet applications, supporting Python 3.6+. It includes modules for many different purposes, including the following:

twisted.web: HTTP clients and servers, HTML templating, and a WSGI server
twisted.conch: SSHv2 and Telnet clients and servers and terminal emulators
twisted.words: Clients and servers for IRC, XMPP, and other IM protocols
twisted.mail: IMAPv4, POP3, SMTP clients and servers
twisted.positioning: Tools for communicating with NMEA-compatible GPS receivers
twisted.names: DNS client and tools for making your own DNS servers
twisted.trial: A unit testing framework that integrates well with Twisted-based code.

Twisted supports all major system event loops -- select (all platforms), poll (most POSIX platforms), epoll (Linux), kqueue (FreeBSD, macOS), IOCP (Windows), and various GUI event loops (GTK+2/3, Qt, wxWidgets). Third-party reactors can plug into Twisted, and provide support for additional event loops.

Installing

To install the latest version of Twisted using pip:

$ pip install twisted

Additional instructions for installing this software are in the installation instructions.

Documentation and Support

Twisted's documentation is available from the Twisted Matrix website. This documentation contains how-tos, code examples, and an API reference.

Help is also available on the Twisted mailing list.

There is also an IRC channel, #twisted, on the Libera.Chat network. A web client is available at web.libera.chat.

Unit Tests

Twisted has a comprehensive test suite, which can be run by tox:

$ tox -l                       # to view all test environments
$ tox -e nocov                 # to run all the tests without coverage
$ tox -e withcov               # to run all the tests with coverage
$ tox -e alldeps-withcov-posix # install all dependencies, run tests with coverage on POSIX platform

You can test running the test suite under the different reactors with the TWISTED_REACTOR environment variable:

$ env TWISTED_REACTOR=epoll tox -e alldeps-withcov-posix

Some of these tests may fail if you:

don't have the dependencies required for a particular subsystem installed,
have a firewall blocking some ports (or things like Multicast, which Linux NAT has shown itself to do), or
run them as root.

Static Code Checkers

You can ensure that code complies to Twisted coding standards:

$ tox -e lint   # run pre-commit to check coding stanards
$ tox -e mypy   # run MyPy static type checker to check for type errors

Or, for speed, use pre-commit directly:

$ pipx run pre-commit run

Copyright

Twisted is made available under the MIT license. The included LICENSE file describes this in detail.

Warranty

THIS SOFTWARE IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE USE OF THIS SOFTWARE IS WITH YOU.

IN NO EVENT WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE LIBRARY, BE LIABLE TO YOU FOR ANY DAMAGES, EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Again, see the included LICENSE file for specific legal details.

txaws's People

Contributors

Stargazers

Watchers

Forkers

gp89 mithrandi markrwilliams exarkun teja624 mrwsr julian xzy3 gmorell dalavancloud moshez diemacht chevah simpoir domamato perfect5th isabella232

txaws's Issues

Bind GetChange for Route 53

The Route 53 API allows polling the status of a change request; this is useful to wait until it has propagated to all the authoritative nameservers.

Support max_keys in the S3 in-memory test double

The real implementation supports it. Add a test to demonstrate the in-memory implementation does as well.

Release 0.3.0

The introduction of Route53 is probably a good reason to do a new release.

Live S3 tests don't handle rate limit errors from AWS

Build https://travis-ci.org/twisted/txaws/builds/190249354 failed:

===============================================================================
[ERROR]
Traceback (most recent call last):
Failure: twisted.internet.defer.FirstError: FirstError[#0, [Failure instance: Traceback: <class 'txaws.s3.exception.S3Error'>: Error Message: Please reduce your request rate.
/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/twisted/web/_newclient.py:1185:_bodyDataFinished_CONNECTED
/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/txaws/client/base.py:146:connectionLost
/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/twisted/internet/defer.py:457:callback
/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/twisted/internet/defer.py:565:_startRunCallbacks
--- <exception caught here> ---
/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/twisted/internet/defer.py:651:_runCallbacks
/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/txaws/s3/client.py:40:s3_error_wrapper
/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/txaws/client/base.py:65:error_wrapper
]]
txaws.s3.tests.test_client.LiveS3TestCase.test_buckets
===============================================================================
[ERROR]
Traceback (most recent call last):
Failure: twisted.trial.util.DirtyReactorAggregateError: Reactor was unclean.
Selectables:
<<class 'twisted.internet.tcp.Client'> to ('s3.amazonaws.com', 443) at 7f0463707cd0>
txaws.s3.tests.test_client.LiveS3TestCase.test_buckets
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/txaws/s3/client.py", line 40, in s3_error_wrapper
    error_wrapper(error, S3Error)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/txaws/client/base.py", line 65, in error_wrapper
    raise fallback_error
txaws.s3.exception.S3Error: Error Message: Please reduce your request rate.
txaws.s3.tests.test_client.LiveS3TestCase.test_objects
-------------------------------------------------------------------------------

Deprecate _CanonicalRequest.from_request_component in favour of something better

This API takes a "sort of decoded" path where the path must be joined on slashes, but the components must not be urlencoded. This generally encourages doing bad things, so we should replace it with something better (eg. a list of decoded path components).

Release 0.4.0

Some cool stuff happened.

Implement the correct deletion restrictions in Route53 _process_delete

txaws/testing/route53.py:

        # The hosted zone itself must always have an SOA and an NS.  It is an                                                                                                                                      
        # error to attempt to delete either of those.                                                                                                                                                              
        #                                                                                                                                                                                                          
        # However, there may be NS records for subdomains of the hosted zone                                                                                                                                       
        # (ie, delegations).  It's okay to delete those.  We're not quite                                                                                                                                          
        # smart enough to recognize that case here.

TLS support includes root certificate management code that should be handled elsewhere

txaws/client/ssl.py is mostly just code that's been obsoleted by improvements to certificate authority certificate handling code in Twisted.

We can probably just get rid of the whole module (after deprecating it, I suppose). Certainly we don't need VerifyingWebClientContext anymore. Instead, we should use BrowserLikePolicyForHTTPS.

This will:

reduce complexity of txaws code (in a security-sensitive area)
bring support for certificate bundles (which txaws can't currently understand)
make Twisted responsible for making sure something good happens on Windows (twisted has Windows CI, txaws does not)

txaws.tests.test_credentials.CredentialsTestCase.test_errors_are_valueerrors_for_backwards_compat

If ~/.aws/credentials exists and has no default profile, txaws.tests.test_credentials.CredentialsTestCase.test_errors_are_valueerrors_for_backwards_compat fails like this:

Traceback (most recent call last):
  File "/home/exarkun/Work/LeastAuthority/txaws/txaws/tests/test_credentials.py", line 36, in test_errors_are_valueerrors_for_backwards_compat
    AWSCredentials(environ={ENV_ACCESS_KEY: "foo"})
  File "/home/exarkun/Environments/txaws/local/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 346, in __exit__
    self._expectedName, reason.getTraceback()),
twisted.trial.unittest.FailTest: txaws.exception.CredentialsNotFoundError raised instead of ValueError:
 Traceback (most recent call last):
  File "/home/exarkun/Environments/txaws/local/lib/python2.7/site-packages/twisted/trial/_asynctest.py", line 112, in _run
    utils.runWithWarningsSuppressed, self._getSuppress(), method)
  File "/home/exarkun/Environments/txaws/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/home/exarkun/Environments/txaws/local/lib/python2.7/site-packages/twisted/internet/utils.py", line 197, in runWithWarningsSuppressed
    result = f(*a, **kw)
  File "/home/exarkun/Work/LeastAuthority/txaws/txaws/tests/test_credentials.py", line 36, in test_errors_are_valueerrors_for_backwards_compat
    AWSCredentials(environ={ENV_ACCESS_KEY: "foo"})
--- <exception caught here> ---
  File "/home/exarkun/Work/LeastAuthority/txaws/txaws/tests/test_credentials.py", line 36, in test_errors_are_valueerrors_for_backwards_compat
    AWSCredentials(environ={ENV_ACCESS_KEY: "foo"})
  File "/home/exarkun/Work/LeastAuthority/txaws/txaws/credentials.py", line 51, in __init__
    _, secret_key = _load_shared_credentials(environ=environ)
  File "/home/exarkun/Work/LeastAuthority/txaws/txaws/credentials.py", line 81, in _load_shared_credentials
    raise CredentialsNotFoundError("No such profile {!r}".format(profile))
txaws.exception.CredentialsNotFoundError: No such profile 'default'

Generate some API docs and host them on readthedocs

So folks can more easily learn about txAWS's features and how to use them.

Use cooperator for hashing

For medium sized objects (let's say larger than 1MB but smaller than 100MB), fitting the object data into memory all at once isn't a big deal, but blocking to calculate the hash will take long enough to significantly impact other things going on in the process, especially if multiple uploads are attempted at once. One possible easy fix for this is to feed data into the hash function in multiple chunks, with cooperator driving the process to allow other servicing other events at the same time.

Support other S3 regions

Currently there is no way to make the S3 client sign requests for another region as the region is hardcoded:

txaws/txaws/s3/client.py

Line 121 in 836b7e5

region=REGION_US_EAST_1,

Support special rrset types in Route53 client

#3 introduced support only for regular resource record sets. change_resource_record_sets and list_resource_record_sets should probably both know about some extra things to fully expose the Route53 functionality: http://docs.aws.amazon.com/Route53/latest/APIReference/API_ResourceRecordSet.html defines the schema for specifying things like "alias resource record sets" and "failover resource record sets" and so on.

I'm just filing one ticket for now but it probably makes sense to have one ticket per special rrset type.

Report errors from Route53 client

Currently Route53 errors result in a no-information Route53Error exception. The error information in the response is dropped on the floor.

Capture the information and put it in the errors attribute.

Add support for SQS

Looks like @lud4ik made a Github repo for implementation of SQS (https://github.com/lud4ik/txAWS/blob/master/txaws/sqs/client.py) . Most of the bits look ok, probably need to redo the signing to use https://github.com/twisted/txaws/blob/master/txaws/_auth_v4.py

travis-ci no longer runs the integration tests

It complains that the environment variables giving the aws credentials are missing.

Release 0.5.0

Some cool stuff happened.

boto3 needs to be installed for S3 cleanup

The Travis config tries to run the cleanup script, but it fails because boto3 is not installed: https://travis-ci.org/twisted/txaws/builds/190452275#L1033

Test failures with Twisted 17.5

17.5 fixed most of our issues, I think, but we're left with these:

[ERROR]
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/txaws/client/tests/test_base.py", line 287, in test_ssl_hostname_verification
    [(host, port, factory, contextFactory)] = fake_reactor.connects
exceptions.ValueError: need more than 0 values to unpack
txaws.client.tests.test_base.BaseQueryTestCase.test_ssl_hostname_verification

[ERROR]
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 992, in startConnectionAttempts
    iterateEndpoint.start(self._attemptDelay)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/twisted/internet/task.py", line 190, in start
    self.starttime = self.clock.seconds()
exceptions.AttributeError: 'FakeReactor' object has no attribute 'seconds'
txaws.ec2.tests.test_client.EC2ClientEBSTestCase.test_attach_volume

The second one is a misattributed traceback, I think.

Bind EC2 GetConsoleOutput

http://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_GetConsoleOutput.html

Implement Auth v4

http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html

Note at the end, "You should use Signature Version 4."

Implement this in a way that's generically usable by all future extensions to txAWS, rather than tied to specific API services like ec2 or s3.

Start producing changelog/news content to inform users of the benefits of new releases

Probably use https://pypi.python.org/pypi/towncrier

AWSServiceEndpoint defaults to not verifying server certificates

Given

$ openssl x509 -in server.pem -noout -text
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number: 1 (0x1)
    Signature Algorithm: md5WithRSAEncryption
        Issuer: OU=example, O=example, CN=example.com, ST=example, C=US/[email protected], L=example
        Validity
            Not Before: Feb 12 00:31:39 2014 GMT
            Not After : Feb 12 00:31:39 2015 GMT
        Subject: OU=example, O=example, CN=example.com, ST=example, C=US/[email protected], L=example
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (1024 bit)
                Modulus:
                    00:a9:11:6a:12:1c:16:9d:15:e3:84:18:a4:ef:6e:
                    7b:b9:a3:64:d6:d9:60:f0:02:f1:c6:a8:40:2d:63:
                    52:02:f8:d8:24:f8:d0:f7:28:3a:26:56:2e:04:64:
                    f9:0c:3a:3f:b3:e8:27:33:bf:4f:8b:9f:2b:4a:a8:
                    3d:42:66:d6:31:2b:0d:15:b2:f5:e5:a5:18:b1:34:
                    82:b3:c4:08:cf:a3:c4:57:cb:5f:12:4b:29:d3:ed:
                    bb:03:00:f0:8a:7b:d8:93:22:ac:53:79:63:f2:53:
                    05:93:91:de:3e:5a:fe:66:d8:e6:d7:11:aa:07:df:
                    2b:19:b8:41:07:a0:9e:18:81
                Exponent: 65537 (0x10001)
    Signature Algorithm: md5WithRSAEncryption
         58:03:4f:a6:9f:7c:e6:75:cc:a0:8c:07:96:c4:8c:9b:c8:a3:
         92:4e:d7:41:4d:e8:8a:80:98:0d:c3:3a:a4:b4:fe:9a:f0:43:
         7b:59:99:73:c9:94:96:b5:b2:66:8f:e5:4f:a1:c5:e3:84:87:
         21:9d:33:0a:03:d8:c1:3a:56:57:a0:b4:2b:ef:e4:2a:56:01:
         11:16:77:6a:3a:f3:19:fe:40:09:f5:13:37:29:39:3f:cd:42:
         eb:c9:9c:f5:f4:3b:96:1a:b5:8a:e3:4a:cd:58:7c:6f:be:3a:
         67:8b:9f:9f:a8:88:d5:5c:a0:9f:23:90:d6:16:7f:94:e5:f9:
         e9:30

and

$ twistd -n web --port ssl:12345:certKey=server.pem
2017-01-09T08:01:57-0500 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 16.4.1 (/usr/bin/python 2.7.12) starting up.
2017-01-09T08:01:57-0500 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.epollreactor.EPollReactor.
2017-01-09T08:01:57-0500 [-] Site (TLS) starting on 12345
2017-01-09T08:01:57-0500 [twisted.web.server.Site#info] Starting factory <twisted.web.server.Site instance at 0x7f3476497cb0>

This txaws-based program:

$ cat testit.py 
from __future__ import print_function

from txaws.client.base import BaseQuery
from txaws.service import AWSServiceEndpoint

from twisted.internet.task import react

def main(reactor):
    endpoint = AWSServiceEndpoint(ssl_hostname_verification=True)
    q = BaseQuery(action=b"GET", endpoint=endpoint, reactor=reactor)
    d = q.get_page(b"https://localhost:12345/")
    d.addCallback(print)
    return d

react(main, [])

produces this result:

$ python testit.py 

            <html>
            <head><title>Twisted Web Demo</title><head>
            <body>
            Hello! This is a Twisted Web test page.
            </body>
            </html>

There is no way the default for txAWS HTTPS requests should be to silently allow communication with a TLS server using such a certificate (self-signed, expired, mismatching hostname).

Respect AWS_PROFILE environment variable

The official AWS SDK will look at AWS_PROFILE and then look up credentials in ~/.aws.

This is a very convenient way to operate and avoids ever putting the sensitive secret material into an environment variable where it might be exposed to an attacker.

Consistently use and require unicode for S3 bucket names

The code is currently fairly sloppy about bytes vs unicode.

Remove the API stability markers

txaws should have the same API compatibility policy as Twisted.

Remove the markers that suggest APIs might change randomly at any time and add compatibility policy documentation pointing at Twisted's docs.

S3 object keys are not quoted

(At least) S3Client.get_object fails to quote the object key into the URL it generates for the request. For any key with anything that's a url special character, this causes problems.

AWS maybe respond to such requests with a weird looking 505 claiming an unsupported HTTP version was requested.

Support the `prefix` parameter accepted by S3 GetBucket

A prefix can be specified to limit the objects which are returned. Support it as an optional argument to get_bucket.

Temporary credentials

You don't have to always use a specific user's AWS credentials to talk to AWS services. Instead, you can use temporary credentials. These come in a couple varieties. I've found the ones made available as part of EC2 IAM roles to be the most useful.

I have code to extract the credentials and security token for a given role from the metadata page that's available to EC2 instances. Because they're temporary, however, they do expire, and the obvious way to implement them allows for a race condition where a request to retrieve a new token happens after a request to a service with an expired token. The correct implementation requires that AWSCredentials gain get_temporary_secret_key and get_temporary_security_token methods that return Deferreds that fire after a refresh request has completed. This will require that callers use maybeDeferred.

It'll be a big change but it makes deploying code to EC2 instances much easier.

Update metadata to point at GitHub

I guess this is the de facto home of txaws now, so we should change the references to Launchpad. (I'll be making a PR for this, just filing the issue so I don't forget)

TXAWS_INTEGRATION_* variables aren't set in a way that makes them usable

The integration tests are failing to find these variables on travis-ci. Not sure why.

Rely on just one XML library

Currently txAWS uses both lxml and the stdlib xml libraries. This is confusing and results in duplication of efforts. Pick one and just use it everywhere.

Support the rest of the Route53 record types

For example, MX. Currently we get a KeyError from the parsing code if any of these happen to exist and one tries to list rrsets.

We should support everything Route53 supports now and we should make the code fail a little more gracefully if new types are added on the AWS side in the future.

get_ca_certs does not dereference symlink

get_ca_certs throws CertsNotFound when /etc/ssl/certs is a symlink. That's because glob.glob doesn't dereference symlinks. The easy solution is to call os.path.realpath on each certificate path before passing it off to glob.glob.

S3 client cannot retrieve objects for keys starting with /

>>> s3_url_context(AWSServiceEndpoint("http://foo/"), bucket="bar", object_name="baz").get_encoded_url()
'http://foo/bar/baz'
>>> s3_url_context(AWSServiceEndpoint("http://foo/"), bucket="bar", object_name="/baz").get_encoded_url()
'http://foo/bar/baz'
>>> s3_url_context(AWSServiceEndpoint("http://foo/"), bucket="bar", object_name="//baz").get_encoded_url()
'http://foo/bar//baz'
>>>

Factoring of _Query causes some duplication of effort

_Query and _CanonicalRequest have both ended up with some URL parsing/generation logic in them. There's even an interaction where _Query goes from a structured URL to a string to pass in to _CanonicalRequest which then parses the string to get back to the structured form (_Query._canonical_request).

Also, the generated headers complicate _Query._canonical_request somewhat. The headers must be generated at the time the request is issued (eg so that the date header is recent). This means the headers used to compute the canonical request can't just be a field on _Query. This leads them to be locals in submit where they're not easily accessible to _canonical_request (leading to them being a parameter to that method instead). This complicates the code and makes the tests somewhat messy (which means it's a pain to write tests that exercise particular pieces of the implementation independently - which is particularly appealing for this code since the ultimate output is a cryptographic hash).

Fix this interface so that:

the code can interact using structured objects to interact.
_canonical_request can be tested without correctly duplicating the set of headers generated by the implementation and passing them in as an argument.

Replace `AWSCredentials()` with some better APIs

Currently it's possible to get a credentials object with filled-out values like this:

from txaws.credentials import AWSCredentials
credentials = AWSCredentials()

This will check the environment:

for AWS_ACCESS_KEY_ID and use it if found
for AWS_SECRET_ACCESS_KEY and use it if found
for AWS_SHARED_CREDENTIALS_FILE and read it if either of the above two are missing
for AWS_PROFILE and use it instead of the "default" section of the credentials file
read the key id or secret key (if either or both were not found in the environment) from the profile in the credentials file

This is a lot of behavior to get from just instantiating the class with no arguments.

I suggest the following APIs instead:

AWSCredentials.from_environment(required_environ_dict) - does the searching that __init__ currently does
AWSCredentials.from_profile(profile_name, optional_credentials_contents) - just looks up the named profile in the given credentials data (read from a credentials file) and initialize from values found there.
AWSCredentials(access_key=..., secret_key=...) - just sets attributes

Most application code will probably want to use the first form.

Special-purpose applications (such as the txaws integration test suite) will probably want the second form (eg `AWSCredentials.from_profile("txaws-integration-tests", ...).

And the last one is a necessary building block for the first two. It's also just good form to have an object behave this way. And it may even be directly useful to some applications that store credentials somewhere other than in the environment or a credentials file.

Tests fail in a weird way if you have an AWS config

I'm assuming it is unintended that this code tries to load real credentials:

[FAIL]
Traceback (most recent call last):
  File "/home/mithrandi/code/txaws/txaws/tests/test_credentials.py", line 36, in test_errors_are_valueerrors_for_backwards_compat
    AWSCredentials(environ={ENV_ACCESS_KEY: "foo"})
  File "/home/mithrandi/deployment/virtualenvs/txaws/site-packages/twisted/trial/_synctest.py", line 346, in __exit__
    self._expectedName, reason.getTraceback()),
twisted.trial.unittest.FailTest: txaws.exception.CredentialsNotFoundError raised instead of ValueError:
 Traceback (most recent call last):
  File "/home/mithrandi/deployment/virtualenvs/txaws/site-packages/twisted/trial/_asynctest.py", line 112, in _run
    utils.runWithWarningsSuppressed, self._getSuppress(), method)
  File "/home/mithrandi/deployment/virtualenvs/txaws/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/home/mithrandi/deployment/virtualenvs/txaws/site-packages/twisted/internet/utils.py", line 197, in runWithWarningsSuppressed
    result = f(*a, **kw)
  File "/home/mithrandi/code/txaws/txaws/tests/test_credentials.py", line 36, in test_errors_are_valueerrors_for_backwards_compat
    AWSCredentials(environ={ENV_ACCESS_KEY: "foo"})
--- <exception caught here> ---
  File "/home/mithrandi/code/txaws/txaws/tests/test_credentials.py", line 36, in test_errors_are_valueerrors_for_backwards_compat
    AWSCredentials(environ={ENV_ACCESS_KEY: "foo"})
  File "/home/mithrandi/code/txaws/txaws/credentials.py", line 51, in __init__
    _, secret_key = _load_shared_credentials(environ=environ)
  File "/home/mithrandi/code/txaws/txaws/credentials.py", line 81, in _load_shared_credentials
    raise CredentialsNotFoundError("No such profile {!r}".format(profile))
txaws.exception.CredentialsNotFoundError: No such profile 'default'


txaws.tests.test_credentials.CredentialsTestCase.test_errors_are_valueerrors_for_backwards_compat

Refactored sigv4 code leaves query arguments off of canonical url to sign

Oops! This causes incorrect signatures for any request with query arguments.

S3Client.head_object is broken

  File "/home/mithrandi/code/txaws/txaws/s3/client.py", line 495, in <lambda>  
    d.addCallback(lambda (response, body): _to_dict(response.responseHeaders)) 
exceptions.AttributeError: 'Response' object has no attribute 'responseHeaders'

The attribute is named headers, not response_headers. Sadly there is no test coverage at all of this code path, so the test suite doesn't catch this.

client.base BaseQuery/query usage?

I know there's some refactoring of the codebase in progress, but what's the recommended pattern for adding a new service, and specifically, what in client.base should be used?

I'd like to use Step Functions with txaws, and I've been poking around at s3.client, ec2.client, and client.base and they all seem to handle communicating with AWS and generating requests/signatures differently.

I've been able to mostly get a sample request working with just client.base.query and the object that it returns - here's a good chunk of my sample code:

from twisted.web.client import FileBodyProducer
from txaws.client.base import BaseClient, BaseQuery, error_wrapper, RequestDetails, query, url_context
import hashlib
from hashlib import sha256

from txaws import _auth_v4
import txaws.credentials as txcreds


# parts removed
reqbody = "{}"
content_sha256 = sha256(reqbody).hexdigest().decode("ascii")
body_producer = FileBodyProducer(BytesIO(reqbody))
amz_headers = {"Target": "AWSStepFunctions.ListStateMachines"}
headers = Headers({"Content-Type": ["application/x-amz-json-1.0"]})
creds = txcreds.AWSCredentials(mykeyid, mysecret)
urlreq = url_context(scheme=u"https", host=u"states.us-east-1.amazonaws.com",port=None, path=[u""],query=[])
details = RequestDetails(region=b"us-east-1", service=b"states", method=b"POST",url_context = urlreq, amz_headers=amz_headers, headers=headers, body_producer=body_producer, content_sha256=content_sha256)
q = query(credentials=creds, details=details)
d = q.submit()
d.ddCallback(cbRequest)

It doesn't quite work - in _Query in client/base.py, there's a call to _auth_v4._CanonicalRequest.from_request_components() with a hardcoded set of a headers_to_sign - unfortunately for me, I need to add x-amz-target to that list, and I think content-type, but that's not a settable option.

Looking at other services, should I completely build my own query factory and not try to reuse anything, or would it be OK to maybe add a parameter to RequestDetails so I can pass in a list of headers to include in the signed headers list, and build a lightweight wrapper around the base query object? Is there a reason that the base query is intentionally not subclassable and instead hidden behind query()?

Thanks for your advice!

txaws.s3.tests.test_client.LiveS3TestCase.test_objects can fail with DirtyReactorAggregateError

On a local test run:

[ERROR]
Traceback (most recent call last):
Failure: twisted.trial.util.DirtyReactorAggregateError: Reactor was unclean.
Selectables:
<<class 'twisted.internet.tcp.Client'> to ('s3.amazonaws.com', 443) at 7fd3e54bb550>

txaws.s3.tests.test_client.LiveS3TestCase.test_objects

This came from the improved-s3-put_object-testing branch which may be relevant but my hunch is that it's caused by a race condition between disconnection and the last Deferred in the test and happens randomly, even on master, with some frequency.

Don't blow up on alias rrsets in list_resource_record_sets

If you list the rrsets in a zone with an alias record:

Traceback (most recent call last):
  File "twisted/web/_newclient.py", line 1185, in _bodyDataFinished_CONNECTED
    self._bodyProtocol.connectionLost(reason)
  File "txaws/client/base.py", line 156, in connectionLost
    d.callback(data)
  File "twisted/internet/defer.py", line 457, in callback
    self._startRunCallbacks(result)
  File "twisted/internet/defer.py", line 565, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File "twisted/internet/defer.py", line 651, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "txaws/route53/client.py", line 242, in _handle_list_resource_record_sets_response
    ttl = int(rrset.find("TTL").text)
exceptions.AttributeError: 'NoneType' object has no attribute 'text'

Supporting these would be one step towards resolving #35.

Grant mithrandi access to the project

As one of the original developers / maintainers (although I haven't been active for quite a while), I'd like to request maintainer access for the GitHub repository :)

Perhaps we should move this to a txaws org? Although it doesn't matter much for now.

Generate APIs from botocore endpoint data

botocore, one of the main building blocks of boto3, the official Python AWS SDK, includes a large pile of JSON which describes the AWS endpoints in rather a lot of detail:

https://github.com/boto/botocore/tree/develop/botocore/data

botocore (or boto3, I'm not really sure) takes this data and uses it to automatically generate many (or most?) of the Python APIs which provide access to the AWS endpoints.

Presumably this saves on quite a bit of implementation complexity and maintenance effort.

Seemingly, txAWS could include a similar generator which creates Twisted-based Python APIs instead.

Would this be a good thing?

Use twisted.web.client.Agent instead of twisted.web.client.getPage

getPage still isn't deprecated! It's amazing. But it really should be and it probably will be. Also Agent is a lot easier to interact with in tests, has certificate validation features almost by default, and is better in a bunch of other ways too.

Implement Route 53 support

I'll be needing this for twisted/txacme#32 so it might as well go in txaws instead of fragmenting the ecosystem further.

The Route53 test double returns unlimited rrsets but should return at most 100

According to https://docs.aws.amazon.com/Route53/latest/APIReference/API_ListResourceRecordSets.html 100 is the maximum number of rrsets. The test double implementation of list_resource_record_sets in txaws/testing/route53.py will return an unlimited number of rrsets if no maxitems is explicitly given, though.

TypeError: init() got an unexpected keyword argument 'credentials'

I've been running into an issue using the S3 client on v0.5.0.

TypeError: __init__() got an unexpected keyword argument 'credentials'
  File "txaws/s3/client.py", line 445, in put_object
    d = self._submit(self._query_factory(details))
  File "txaws/s3/client.py", line 82, in _query_factory
    return self.query_factory(credentials=self.creds, details=details, **kw)
  File "txaws/s3/client.py", line 692, in __init__
    super(Query, self).__init__(*args, **kwargs)

It seems query_factory method in the client is attempting to pass a credentials kwarg to the BaseQuery object, however it expects a creds kwarg instead.

txaws/txaws/s3/client.py

Lines 81 to 82 in 5c33173

 def _query_factory(self, details, **kw): 

 return self.query_factory(credentials=self.creds, details=details, **kw)

txaws/txaws/client/base.py

Lines 633 to 635 in 5c33173

 class BaseQuery(object): 

 def __init__(self, action=None, creds=None, endpoint=None, reactor=None,

PR here if it helps: #98

Replace attrs.assoc with attrs.evolve

Looks like assoc is deprecated in favor of evolve. However, evolve is only available as of 17.1.0, so we would have to pin attrs >= 17.1.0 in setup.py.

	def _query_factory(self, details, **kw):
	return self.query_factory(credentials=self.creds, details=details, **kw)

	class BaseQuery(object):

	def __init__(self, action=None, creds=None, endpoint=None, reactor=None,