Comments (2)
From further digging, it seems like the data is being serilized twice before upload.
I checked this by re-running the code above, then also running a print
of
n = Item().data.serialize(Item().data.serialize(dict(hello="message", nested={"hello": "message"}, enabled=False, number=0)))
print(n)
>>
{'hello': {'M': {'S': {'S': 'message'}}}, 'nested': {'M': {'M': {'M': {'hello': {'M': {'S': {'S': 'message'}}}}}}}, 'enabled': {'M': {'BOOL': {'BOOL': False}}}, 'number': {'M': {'N': {'N': 0}}}}
Looking at the tests for map attribute, it's all based on mock data but has is there any tests of data that is stored in Dynamo then retrieved again?
will keep digging
from falcano.
And a bit more digging -
Looking at the api call to put_item, even though serialize
is defaulted to false, it looks like Map Attributes are still serialized
If I print out kwargs that are passed to put_item under the Model
class, i see this:
new = Item(hash_key=f'testItem#{id}', range_key="testRange#3749189f-3d4c-4937-b31a-924bc84b366c",
data=dict(hello="message", nested={"hello": "message"}, enabled=False, number=0)
)
print(new.save())
>>>
{'TableName': 'my-table', 'Item': {'Type': 'testItem', 'data': {'hello': {'S': 'message'}, 'nested': {'M': {'hello': {'S': 'message'}}}, 'enabled': {'BOOL': False}, 'number': {'N': 0}}, 'pk': 'testItem#<built-in function id>', 'sk': 'testRange#3749189f-3d4c-4937-b31a-924bc84b366c'}, 'ReturnValues': 'NONE'}
The issue seems to be that the put_item API for the Table Resource doesn't require any serialization and does it automatically.
I tested this by running the API calls directly with my raw input and it stores as expected in Dynamo
import boto3
table = boto3.resource('dynamodb')
table = table.Table('my-table')
table.put_item(Item={'Type': 'testItem', 'data': dict(hello="message", nested={"hello": "message"}, enabled=False, number=0), 'pk': 'testItem#cca06093-e0c6-49a6-93d5-266bed23cad1', 'sk': 'testRange#3749189f-3d4c-4937-b31a-924bc84b366c'})
The result is a successful API call and the Dynamo console shows the expected outcome:
{
"pk": {
"S": "testItem#cca06093-e0c6-49a6-93d5-266bed23cad1"
},
"sk": {
"S": "testRange#3749189f-3d4c-4937-b31a-924bc84b366c"
},
"Type": {
"S": "testItem"
},
"data": {
"M": {
"number": {
"N": "0"
},
"hello": {
"S": "message"
},
"nested": {
"M": {
"hello": {
"S": "message"
}
}
},
"enabled": {
"BOOL": false
}
}
}
}
Fix appears to stop whatever serilizes the MapAttribute before put_item is called as this is an unnecessary step for the current API choice - Either that or change the API from the Resource Table.put_item to the DynamoDB Client put_item, which does need serialization - however this would mean all values need to be serialized, not just Map.
I'll see if I can fix and submit a PR
from falcano.
Related Issues (6)
- BooleanAttribute Missing HOT 2
- ListAttribute can only be a list of MapAttribute HOT 2
- Unknown parameter in input: "IndexName", must be one of: TableName, Key, AttributesToGet, ConsistentRead, ReturnConsumedCapacity, ProjectionExpression, ExpressionAttributeNames HOT 1
- UnicodeSetAttribute sorts my set HOT 5
- Set minimum version to 3.8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from falcano.