stevearc / dynamo3 Goto Github PK

https://blog.travis-ci.com/oss-announcement

While testing with Flywheel and moto on Python 2.7, I found what seems to be an incompatibility. I think the problem is in dynamo3, but it's difficult to see a fix without potentially breaking existing data (I think!). Note that I haven't yet tested this on real DynamoDB - just using moto. However, the dynamo3 code looks incorrect to me at first glance.

The issue seems to be that while Flywheel correctly interprets a str value as binary during the save (generating a 'B' Dynamo value), the get path involves dynamo3's value encoding path, which seems to incorrectly encode str as a STRING, not a BINARY.

Here's a repro script:

from flywheel import Engine, Model, Field
from moto import mock_dynamodb2


class Thing(Model):
    id = Field(type=str, hash_key=True)


@mock_dynamodb2
def go():
    engine = Engine()
    engine.connect('dummy-region')
    engine.register(Thing)
    engine.create_schema()

    thing_id='hello'
    thing = Thing(id=thing_id)
    engine.save([thing])

    loaded = engine.get(Thing, [thing.id])
    assert len(loaded) > 0

if __name__ == '__main__':
    go()

Here are the requirements I have in my Python venv:

botocore==1.4.93
docutils==0.13.1
-e [email protected]:danfairs/dynamo3.git@012f2d95649b8c942e0df930351809235200d6d6#egg=dynamo3
flywheel==0.5.1
httpretty==0.8.10
Jinja2==2.9.4
jmespath==0.9.0
MarkupSafe==0.23
moto==0.4.30
python-dateutil==2.6.0
pytz==2016.10
requests==2.12.4
six==1.10.0
Werkzeug==0.11.15
xmltodict==0.10.2

(The fork of dynamo3 has no changes from your master, at the time of writing)

What appears to be the offending lines of code are here:

This doesn't make sense to me: if the user were genuinely storing binary data in a value, it doesn't make sense to do UTF8 decoding on it (what if it were an image, for example?).

After the ConsumedCapacity fixes in #2, I'm seeing the following exception being thrown from https://github.com/stevearc/dynamo3/blob/master/dynamo3/batch.py#L280

'ConsumedCapacity' object is not iterable

I'm guessing it's a result of something tripping https://github.com/stevearc/dynamo3/blob/master/dynamo3/connection.py#L246 before the batch operation on the same connection.

Sending anything other than {"ReturnConsumedCapacity": "NONE"} on Batch* operations will cause an exception:

# dynamo3/result.py:29-30

cap = data.get('ConsumedCapacity', {})
self.capacity += cap.get('CapacityUnits', 0)  # AttributeError·'list' object has no attribute 'get

This is because these operations return ConsumedCapacity as a list due to the operation potentially affecting many tables. For example:

"ConsumedCapacity": [
        {
            "TableName": "Forum",
            "CapacityUnits": 3
        },
        {
            "TableName": "Thread",
            "CapacityUnits": 1
        }
    ]

See http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchGetItem.html