API Endpoint expects trailing slash due to URL Join

We should check that the customer has provided a well-structure Tinybird API endpoint. A regex should suffix.

Hash in PK causes string match failure

A DynamoDB Key may have a # as a field separator in the String, this causes strange behaviour when that field is used in WHERE clauses.

DynamoDB character set to Tinybird character set mapping

DynamoDB allows - in Table names, but Tinybird does not allow them in Datasource Names. We should enforce character remapping when re-using names from source systems.

Relying on user setting DynamoDB Table name in the PITR Export is brittle.

We may be able to extract the name reliably from the export manifest, but should move away from relying on it being in the s3 key.

Issue

The error "Object of type set is not JSON serializable" occurs because the JSON encoder does not know how to handle sets, which can be used in JSON objects (under the "SS" field) in the https://github.com/tinybirdco/DynamoDBExporter/blob/main/DDBStreamCDC/lambda_function.py function.

JSON export fragment example

"companyAssociations": {
    "SS": [
        "01FW81573Y8GAP9ACYHXJK6RYT",
        "01FW81573YVZP29K5RZWJG02EM"
    ]
},

Solution

To handle this it's required to extend the custom JSON encoder to also handle sets by converting them to lists, which are JSON serializable.

import json
from decimal import Decimal
from boto3.dynamodb.types import Binary
import base64

class DDBEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, Decimal):
            return float(o)
        elif isinstance(o, Binary):
            return base64.b64encode(o.value).decode('utf-8')
        elif isinstance(o, bytes):
            return base64.b64encode(o).decode('utf-8')
        elif isinstance(o, set):
            return list(o)
        return super().default(o)

Testing the code with a customer and came across two issues:

• When using a non default AWS Region, there are lambda errors with the file manifest, and no transfers are made to Tinybird.
• There needs to be a confirmation that the S3 bucket and the Lambda function are in the same AWS Region.