Giter Club home page Giter Club logo

dynamodb-data-mapper-js's Introduction

Amazon DynamoDB DataMapper For JavaScript

Apache 2 License

This repository hosts several packages that collectively make up an object to document mapper for JavaScript applications using Amazon DynamoDB.

Getting started

The @aws/dynamodb-data-mapper package provides a simple way to persist and load an application's domain objects to and from Amazon DynamoDB. When used together with the decorators provided by the @aws/dynamodb-data-mapper-annotations package, you can describe the relationship between a class and its representation in DynamoDB by adding a few decorators:

import {
    attribute,
    hashKey,
    rangeKey,
    table,
} from '@aws/dynamodb-data-mapper-annotations';

@table('table_name')
class MyDomainObject {
    @hashKey()
    id: string;

    @rangeKey({defaultProvider: () => new Date()})
    createdAt: Date;

    @attribute()
    completed?: boolean;
}

With domain classes defined, you can interact with records in DynamoDB via an instance of DataMapper:

import {DataMapper} from '@aws/dynamodb-data-mapper';
import DynamoDB = require('aws-sdk/clients/dynamodb');

const mapper = new DataMapper({
    client: new DynamoDB({region: 'us-west-2'}), // the SDK client used to execute operations
    tableNamePrefix: 'dev_' // optionally, you can provide a table prefix to keep your dev and prod tables separate
});

Supported operations

Using the mapper object and MyDomainObject class defined above, you can perform the following operations:

put

Creates (or overwrites) an item in the table

const toSave = Object.assign(new MyDomainObject, {id: 'foo'});
mapper.put(toSave).then(objectSaved => {
    // the record has been saved
});

get

Retrieves an item from DynamoDB

mapper.get(Object.assign(new MyDomainObject, {id: 'foo', createdAt: new Date(946684800000)}))
    .then(myItem => {
        // the item was found
    })
    .catch(err => {
        // the item was not found
    })

NB: The promise returned by the mapper will be rejected with an ItemNotFoundException if the item sought is not found.

update

Updates an item in the table

const myItem = await mapper.get(Object.assign(
    new MyDomainObject,
    {id: 'foo', createdAt: new Date(946684800000)}
));
myItem.completed = true;

await mapper.update(myItem);

delete

Removes an item from the table

await mapper.delete(Object.assign(
    new MyDomainObject,
    {id: 'foo', createdAt: new Date(946684800000)}
));

scan

Lists the items in a table or index

for await (const item of mapper.scan(MyDomainObject)) {
    // individual items will be yielded as the scan is performed
}

// Optionally, scan an index instead of the table:
for await (const item of mapper.scan(MyDomainObject, {indexName: 'myIndex'})) {
    // individual items will be yielded as the scan is performed
}

query

Finds a specific item (or range of items) in a table or index

for await (const foo of mapper.query(MyDomainObject, {id: 'foo'})) {
    // individual items with a hash key of "foo" will be yielded as the query is performed
}

Batch operations

The mapper also supports batch operations. Under the hood, the batch will automatically be split into chunks that fall within DynamoDB's limits (25 for batchPut and batchDelete, 100 for batchGet). The items can belong to any number of tables, and exponential backoff for unprocessed items is handled automatically.

batchPut

Creates (or overwrites) multiple items in the table

const toSave = [
    Object.assign(new MyDomainObject, {id: 'foo', completed: false}),
    Object.assign(new MyDomainObject, {id: 'bar', completed: false})
];
for await (const persisted of mapper.batchPut(toSave)) {
    // items will be yielded as they are successfully written
}
batchGet

Fetches multiple items from the table

const toGet = [
    Object.assign(new MyDomainObject, {id: 'foo', createdAt: new Date(946684800000)}),
    Object.assign(new MyDomainObject, {id: 'bar', createdAt: new Date(946684800001)})
];
for await (const found of mapper.batchGet(toGet)) {
    // items will be yielded as they are successfully retrieved
}

NB: Only items that exist in the table will be retrieved. If a key is not found, it will be omitted from the result.

batchDelete

Removes multiple items from the table

const toRemove = [
    Object.assign(new MyDomainObject, {id: 'foo', createdAt: new Date(946684800000)}),
    Object.assign(new MyDomainObject, {id: 'bar', createdAt: new Date(946684800001)})
];
for await (const found of mapper.batchDelete(toRemove)) {
    // items will be yielded as they are successfully removed
}

Operations with Expressions

Aplication example
import {
    AttributePath,
    FunctionExpression,
    UpdateExpression,
} from '@aws/dynamodb-expressions';

const expr = new UpdateExpression();

// given the anotation bellow
@table('tableName')
class MyRecord {
    @hashKey()
    email?: string;

    @attribute()
    passwordHash?: string;

    @attribute()
    passwordSalt?: string;

    @attribute()
    verified?: boolean;

    @attribute()
    verifyToken?: string;
}

// you make a mapper operation as follows
const aRecord = Object.assign(new MyRecord(), {
    email,
    passwordHash: password,
    passwordSalt: salt,
    verified: false,
    verifyToken: token,
});
mapper.put(aRecord, { 
    condition: new FunctionExpression('attribute_not_exists', new AttributePath('email') 
}).then( /* result handler */ );

Table lifecycle operations

createTable

Creates a table for the mapped class and waits for it to be initialized:

mapper.createTable(MyDomainObject, {readCapacityUnits: 5, writeCapacityUnits: 5})
    .then(() => {
        // the table has been provisioned and is ready for use!
    })
ensureTableExists

Like createTable, but only creates the table if it doesn't already exist:

mapper.ensureTableExists(MyDomainObject, {readCapacityUnits: 5, writeCapacityUnits: 5})
    .then(() => {
        // the table has been provisioned and is ready for use!
    })
deleteTable

Deletes the table for the mapped class and waits for it to be removed:

await mapper.deleteTable(MyDomainObject)
ensureTableNotExists

Like deleteTable, but only deletes the table if it exists:

await mapper.ensureTableNotExists(MyDomainObject)

Constituent packages

The DataMapper is developed as a monorepo using lerna. More detailed documentation about the mapper's constituent packages is available by viewing those packages directly.

dynamodb-data-mapper-js's People

Contributors

acheronfail avatar adamclerk avatar aregier avatar brandonl avatar carsonbradshaw avatar hassankhan avatar jeskew avatar jupiter avatar lucasbadico avatar ofekray avatar tjwebb avatar winjer avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dynamodb-data-mapper-js's Issues

Can a field be configured to always update on save?

I'm using the mapper with this schema based aprouch.

Object.defineProperties(MyDomainModel.prototype, {
    [DynamoDbSchema]: {
        value: {
            phone: {
                type: 'Dictionary',
                memberType: embed(Phone)
            },
        },
    },
}); 

createdAt and updatedAt

I see that for the attr createdAt I can send a defaultProvider instruction with new Date(). In the same way to the v4 for generating a uuid.

There is a way to send a instruction for the mapper change the attr updatedAt every time? Like a forced provider.

Object.defineProperties(MyDomainModel.prototype, {
    [DynamoDbSchema]: {
        value: {
            createdAt: {
                type: 'Date',
                defaultProvider: () => new Date()
            },
        },
    },
}); 

version attribute

class MyDomainClass {
    @autoGeneratedHashKey()
    id: string;

    @rangeKey({defaultProvider: () => new Date()})
    createdAt: Date;

    @versionAttribute()
    version: number;
}

How to use this version attribute in the Object.defineProperties way?

error TS2420: Class 'ObjectSet<T>' incorrectly implements interface 'Set<T>'. Property 'toJSON' is missing in type 'ObjectSet<T>'.

I am getting the below error when I compile a typescript file which includes @aws/dynamodb-data-mapper-annotations

$ node_modules/typescript/bin/tsc
node_modules/@aws/dynamodb-auto-marshaller/build/ObjectSet.d.ts(1,31):
error TS2420: Class 'ObjectSet<T>' incorrectly implements interface 'Set<T>'.
  Property 'toJSON' is missing in type 'ObjectSet<T>'.

My system setup is ...

$ node -v
v6.10.3
$ node_modules/typescript/bin/tsc --version
Version 2.7.2

package.json

{
  "name": "typescript-example",
  "version": "0.0.0",
  "description": "",
  "main": "index.js",
  "keywords": [],
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "tslint": "^5.5.0",
    "tslint-config-standard": "^6.0.1",
    "typedoc": "^0.11.0"
  },
  "dependencies": {
    "@aws/dynamodb-data-mapper-annotations": "^0.4.0",
    "@types/node": "^8.10.9",
    "aws-sdk": "^2.7.0"
  }
}

This is the only typescript file in the project (Org.ts)

import {
    attribute,
    autoGeneratedHashKey,
    rangeKey,
    table,
} from '@aws/dynamodb-data-mapper-annotations';

@table('orgs')
export class Org {
    @autoGeneratedHashKey()
    id: string;

    @rangeKey()
    createdAt: Date;

    @attribute()
    authorUsername: string;

    @attribute()
    title: string;
}

About update Expressions

Update expressions allow the partial, in place update of a record in DynamoDB. The expression may have up to four clauses, one containing directives to set values in the record, one containing directives to remove attributes from the record, one containing directives to add values to a set, and the last containing directives to delete values from a set.

A set is just a Set or could also be a List ?

DataMapper batchGet method example

Hello, sorry for opening an issue but I just need to know how to use the batchGet in the Data Mapper.

I created the model classes using DataMapper Annotations and passed new instances of those classes to the batchGet method in a for await but it returns null.

These classes work fine for other methods such as get, put etc.

Example

const gets = [new Class1(), new Class2()];

for await (const item of mapper.batchGet(gets)) {
                console.log(item)
}

Thank you.

Issues upgrading to 0.7.0 release

Hi,

Busy testing dynamodb-data-mapper-js. Been working well with my serverless aws lambda test project but when I upgrade from 0.6.1 to 0.7.0 I get a lot of errors from webpack when building. Below is a small sample of the errors.

ERROR in ./node_modules/@aws/dynamodb-data-mapper/build/index.mjs 1171:71-84
Can't import the named export 'AttributePath' from non EcmaScript module (only default export is available)
 @ ./functions/business/create.ts
 @ multi ./source-map-install.js ./functions/business/create.ts

ERROR in ./node_modules/@aws/dynamodb-data-mapper/build/index.mjs 1182:47-60
Can't import the named export 'AttributePath' from non EcmaScript module (only default export is available)
 @ ./functions/business/create.ts
 @ multi ./source-map-install.js ./functions/business/create.ts

ERROR in ./node_modules/@aws/dynamodb-data-marshaller/build/index.mjs 261:8-21
Can't import the named export 'AttributePath' from non EcmaScript module (only default export is available)
 @ ./node_modules/@aws/dynamodb-data-mapper/build/DataMapper.js
 @ ./node_modules/@aws/dynamodb-data-mapper/build/index.js
 @ ./node_modules/@aws/dynamodb-data-mapper-annotations/build/index.mjs
 @ ./models/business.ts
 @ ./functions/business/create.ts
 @ multi ./source-map-install.js ./functions/business/create.ts

ERROR in ./node_modules/@aws/dynamodb-data-marshaller/build/index.mjs 241:47-60
Can't import the named export 'AttributePath' from non EcmaScript module (only default export is available)
 @ ./node_modules/@aws/dynamodb-data-mapper/build/DataMapper.js
 @ ./node_modules/@aws/dynamodb-data-mapper/build/index.js
 @ ./node_modules/@aws/dynamodb-data-mapper-annotations/build/index.mjs
 @ ./models/business.ts
 @ ./functions/business/create.ts
 @ multi ./source-map-install.js ./functions/business/create.ts

ERROR in ./node_modules/@aws/dynamodb-data-marshaller/build/index.mjs 239:38-51
Can't import the named export 'AttributePath' from non EcmaScript module (only default export is available)
 @ ./node_modules/@aws/dynamodb-data-mapper/build/DataMapper.js
 @ ./node_modules/@aws/dynamodb-data-mapper/build/index.js
 @ ./node_modules/@aws/dynamodb-data-mapper-annotations/build/index.mjs
 @ ./models/business.ts
 @ ./functions/business/create.ts
 @ multi ./source-map-install.js ./functions/business/create.ts

ERROR in ./node_modules/@aws/dynamodb-data-marshaller/build/index.mjs 89:19-32
Can't import the named export 'AttributePath' from non EcmaScript module (only default export is available)
 @ ./node_modules/@aws/dynamodb-data-mapper/build/DataMapper.js
 @ ./node_modules/@aws/dynamodb-data-mapper/build/index.js
 @ ./node_modules/@aws/dynamodb-data-mapper-annotations/build/index.mjs
 @ ./models/business.ts
 @ ./functions/business/create.ts
 @ multi ./source-map-install.js ./functions/business/create.ts

...

emitDecoratorMetadata tsconfig setting causes objects in property arrays to be empty

v0.6.1 (unable to use v0.7.0)

If I set emitDecoratorMetadata=true in tsconfig, then all my properties (attributes) that are arrays of objects will be persisted as an empty Javascript object in an array in the Dynamo DB table. Setting emitDecoratorMetadata=false resolves this issue.

    @table('business')
    export class Business {
        @autoGeneratedHashKey()
        id: string;

        @attribute()
        businessName: string;

        @attribute()
        primaryContact: Contact;

        @attribute({memberType: embed(Account)})
        accounts: Array<Account>;
    }

Accounts property persisted as:
"accounts":[{}]

Question about general future roadmap

Hi i'm the maintainer of DynamoDB-TypeORM, and it seems to happen what you guys building here is pretty similar.
As an big fan and customer of AWS, we'd love to see you guys providing tools for us, but also wanna make sure how you guys gonna develop this
Can you checkout work on our side and share your future plans about dynamodb-data-mapper-js? it would really helpful us to decide how we gonna invest our resources.

Error installing package @aws/dynamodb-data-mapper

c:\Code>npm i @aws/dynamodb-data-mapper --save-dev
npm ERR! code ETARGET
npm ERR! notarget No matching version found for @aws/dynamodb-auto-marshaller@^0
.0.0
npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! notarget a package version that doesn't exist.
npm ERR! notarget
npm ERR! notarget It was specified as a dependency of '@aws/dynamodb-expressions
'
npm ERR! notarget

npm ERR! A complete log of this run can be found in:


1 verbose cli 'i',
1 verbose cli '@aws/dynamodb-data-mapper',
1 verbose cli '--save-dev' ]
2 info using [email protected]
3 info using [email protected]
4 verbose npm-session 00f8b4978f16d8bc
5 silly install loadCurrentTree
6 silly install readLocalPackageData
7 http fetch GET 304 https://registry.npmjs.org/@aws%2fdynamodb-data-mapper 1617ms (from cache)
8 silly pacote tag manifest for @aws/dynamodb-data-mapper@latest fetched in 1646ms
9 silly install loadIdealTree
10 silly install cloneCurrentTreeToIdealTree
11 silly install loadShrinkwrap
12 silly install loadAllDepsIntoIdealTree
13 silly resolveWithNewModule @aws/[email protected] checking installable status
14 http fetch GET 304 https://registry.npmjs.org/@aws%2fdynamodb-expressions 320ms (from cache)
15 silly pacote range manifest for @aws/dynamodb-expressions@^0.1.0 fetched in 323ms
16 silly resolveWithNewModule @aws/[email protected] checking installable status
17 http fetch GET 304 https://registry.npmjs.org/@aws%2fdynamodb-data-marshaller 572ms (from cache)
18 silly pacote range manifest for @aws/dynamodb-data-marshaller@^0.1.0 fetched in 575ms
19 silly resolveWithNewModule @aws/[email protected] checking installable status
20 http fetch GET 304 https://registry.npmjs.org/@aws%2fdynamodb-auto-marshaller 403ms (from cache)
21 silly pacote range manifest for @aws/dynamodb-auto-marshaller@^0.1.0 fetched in 405ms
22 silly resolveWithNewModule @aws/[email protected] checking installable status
23 silly registry:manifest no matching version for @aws/dynamodb-auto-marshaller@^0.0.0 in the cache. Forcing revalidation
24 http fetch GET 304 https://registry.npmjs.org/@aws%2fdynamodb-auto-marshaller 341ms (from cache)
25 silly fetchPackageMetaData error for @aws/dynamodb-auto-marshaller@^0.0.0 No matching version found for @aws/dynamodb-auto-marshaller@^0.0.0
26 verbose type range
27 verbose stack @aws/dynamodb-auto-marshaller: No matching version found for @aws/dynamodb-auto-marshaller@^0.0.0

Example for updateItem with Increment

Hi,

I cant figure out how to get e.g.:
"UpdateExpression": "set Replies = Replies + :num",
in a call to UpdateItem via the update method. Can you give me a clue?

Detecting ItemNotFoundException

It's a little unclear to me how to detect that dynamo.get() is throwing an ItemNotFoundException. In my code I'm trying this:

try {
  const response = await dynamo.get(param)
  return response
} catch (err) {
  if (err instanceof ItemNotFoundException) {
    throw new Error('[404] Not Found')
  } else {
    throw err
  }
}

Here instanceof never detects that the thrown error is ItemNotFoundException (returns false). Is there some other way to do it, or am I missing something in TypeScript options?

As an alternative, perhaps ItemNotFoundException should include a .code attribute like AWS SDK errors do. Then you could easily check if (err.code === 'ItemNotFoundException') to handle the error.

PS. This is under Serverless Framework + serverless-plugin-typescript.

Is there a way to get the LastEvaluatedKey?

Hey!

I'm implementing a regular in-browser pagination: list 10 items, click next, go to the next page.

For paginating to the next page I need to set the startKey with the lastEvaluatedKey from the previous page. I'm currently, generating the lastEvaluatedKey programmatically, since I know what are the keys, but I'm wondering if there's a way to retrieve it from the query result, since the SDK returns it.

Is there a way to get the lastEvaluatedKey from the query result?

DataMapper batchPut method example

Trying to get my head around how to successfully use this to batch insert records, but keep ending with error messages - I assume what I input in the batchPut function is in the wrong format.

Would it be possible to provide some example?

Thanks

KeySchema.d.ts (10,30): Cannot find name 'KeyType'

When deploying using Serverless Framework 1.26.1 and serverless-plugin-typescript 1.1.5, I get this error when using dynamodb-data-mapper 0.4.0 and dynamodb-data-mapper-annotations 0.4.0:

node_modules/@aws/dynamodb-data-marshaller/build/KeySchema.d.ts (10,30): Cannot find name 'KeyType'.

At runtime, in the Lambda function, I see another error when executing:

module initialization error: Error
at /var/task/node_modules/@aws/dynamodb-data-mapper-annotations/build/attribute.js:70:19
at DecorateProperty (/var/task/node_modules/reflect-metadata/Reflect.js:553:33)
at Object.decorate (/var/task/node_modules/reflect-metadata/Reflect.js:123:24)
at __decorate (/var/task/lib/tables.js:4:92)
at /var/task/lib/tables.js:17:5
at Object.<anonymous> (/var/task/lib/tables.js:27:2)
at Module._compile (module.js:570:32)
at Object.Module._extensions..js (module.js:579:10)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)

When I downgrade to dynamodb-data-mapper 0.3.2 and dynamodb-data-mapper-annotations 0.3.2, everything works fine.

Error: '...No object property was found at the `DynamoDbSchema` symbol'

FYI the full error is:

The provided item did not adhere to the DynamoDbDocument protocol. No object property was found at the DynamoDbSchema symbol

I'm receiving this error at runtime based on the following setup:

Base class:

import { attribute } from '@aws/dynamodb-data-mapper-annotations';

export class BaseDataObject {
  @attribute()
  createdDate: string;

  @attribute()
  createdBy: string;

  @attribute()
  updatedDate: string;

  @attribute()
  updatedBy: string;
}

Domain class:

import { autoGeneratedHashKey, table } from '@aws/dynamodb-data-mapper-annotations';
import { BaseDataObject } from './base-data-object';

@table('DOMAIN_TABLE')
export class SomeDomainObject extends BaseDataObject {

  @autoGeneratedHashKey()
  id: string;

}

DataMapper execution:

export async function handler(event: any, context: Context): Promise<LambdaResponse> {
  const dynamoInstance: DynamoDB = new AWS.DynamoDB({
    region: 'us-west-2'
  });
  const dataMapper: DataMapper = new DataMapper({
    client: dynamoInstance
  });

  log.verbose('Attempting to write a record...');

  let userId: string = 'Not Provided';
  try {
    if (event.headers.Authorization) {
      const decodedInfo: any = JwtDecoder(event.headers.Authorization);
      userId = decodedInfo.sub.replace(/^[A-z0-9]*\|{1}/, '');
    }
  } catch (e) {
    log.error('Failed to parse the provided JWT');
    return handleError(e);
  }

  const payload: SomeDomainObject = JSON.parse(event.body);
  payload.createdDate = new Date().toISOString();
  payload.createdBy = userId;

  return dataMapper.put(payload).then(handleSuccess, handleError);
}

Any ideas?

Read all results in one call (for API responses)

The for async syntax is a nice way to hide lastEvaluatedKey/exclusiveStartKey. But when returning data from an API, it involves a clumsy need to do something like:

const items = []
for async (const item of dynamo.query(...)) {
  items.push(item)
}
return items

So I'm wondering if it would be possible to add some simple way to read all the results using a single function call? Something like:

return await readAll(dynamo.query())

I am also slightly worried about the performance implications of async iterators, if every item is resolved as a separate promise?

Btw, overall I am really excited about the whole dynamodb-data-mapper library. Having written a lot of code using AWS.DynamoDB.DocumentClient, I appreciate how this library solves many painful things in it. ๐Ÿ‘

Batch operations

The data mapper does not have any methods defined that map to BatchGetItem and BatchWriteItem. This should be added.

Query/Scan limit is ignored

I can specify the limit parameter to query() and scan(), but because the result iterator keeps reading on using lastEvaluatedKey, all result rows are always returned and limit only becomes a batch size parameter.

Perhaps the iterator should manually check that if limit is reached, it stops yielding results?

Query pagination

I'm having a little difficulty understanding how to paginate and was hoping you could clarify.

Given the following code to pull 2 releases from a release group, I seem to be getting all releases in the group instead.

const iterator = mapper.query(Release, { group }, { pageSize: 2 })
const releases = []
for await (const release of iterator) {
  releases.push(release)
}
return releases

I tried also using the deprecated limit option but no joy.

Perhaps this is to do with the async iterator, does it auto paginate under the hood? Any help appreciated.

[Edit] I see #7 talks a little about this, so would appear my assumption about auto pagination may be correct.

So setting a pageSize of 2 and then breaking from the loop after 2 iterations should mean only a single query hitting dynamo. I'll close the issue but please comment if this assumption is off the mark.

Creating and Dropping a Table

I've been trialling this new mapper out and very happy with how easy it was to get up and running.

One issue I hit was having to create tables in the more verbose style. This felt like I was repeating many of the decorated values of the classes defining my model for this mapper.

Are there plans to support a table create and drop operation?

Global Secondary Index

  1. Can I use this mapper to create DynamoDB tables?
  2. If so, how do I create global secondary indexes and specify rcu/wcu and other parameters?

asyncIterator & Decorators incompatible with AWS Lambda Node v8.10

Maybe I'm missing something, but I seem to be stuck between a rock and a hard place here.

I'm trying to handle a DynamoDB Stream event which requires a batchGet operation. This requires the experimental async iterator which has been creating issues when I deploy my handler.

If I only use the TypeScript compiler and polyfill then I get this error:

Syntax error in module 'lib/functions/load-audience-trigger': SyntaxError
for await (const audience of dataMapper.batchGet(mappedAudiences)) {
^^^^^

SyntaxError: Unexpected reserved word
at createScript (vm.js:80:10)
at Object.runInThisContext (vm.js:139:10)
at Module._compile (module.js:616:28)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)

Since the for await ... of loop isn't officially supported yet and I can't execute the lambda with --harmony_async_iteration Node flag, I tried to downgrade to a Node v8.10 target using Babel. However this introduced a new problem where the experimental TS decorators used in the DDB entity classes cause the lambda to fail before it can even be imported:

module initialization error
Error

I assume this is a result of further manipulation by Babel after the TypeScript compiler has done it's work. However, without Babel I can't get beyond the syntax error generated around the for await ... of loop.

Are there any examples of how the data mapper can be used in the context of a DDB stream handler?

ItemNotFoundException

mapper.get throws an ItemNotFoundException - except whatever it throws looks kind of broken:

     { Error
         at new ItemNotFoundException (/app/node_modules/@aws/dynamodb-data-mapper/build/ItemNotFoundException.js:12:28)
         at DataMapper.<anonymous> (/app/node_modules/@aws/dynamodb-data-mapper/build/DataMapper.js:659:31)
         at step (/app/node_modules/tslib/tslib.js:133:27)
         at Object.next (/app/node_modules/tslib/tslib.js:114:57)
         at fulfilled (/app/node_modules/tslib/tslib.js:104:62)
         at <anonymous>
         at process._tickCallback (internal/process/next_tick.js:160:7)
       itemSought:
        { TableName: 'charts',
          Key: { owner: [Object], fullname: [Object] },
          ConsistentRead: false } }

I suspect somewhere a function is being returned instead of called?

Exclude tsconfig.json from build output

I get the following in VSCode:
No inputs were found in config file '/Users/ralf/code/pitta/ts/node_modules/@aws/dynamodb-auto-marshaller/tsconfig.json'. Specified 'include' paths were '["**/*"]' and 'exclude' paths were '["./build"]'.

This is the same for all of the packages. The build has a tsconfig.json in the output but no Typescript files. Can you please exclude tsconfig.json from the individual packages?

Support for dynamoDB doc client?

Would be really useful. Currently it is not compatible:

TypeError: this.client.batchGetItem is not a function

docClient has only batchGet

How to handle nullable attributes?

I'm having a DynamoDB property which can have value type String or Null. My TypeScript mapping looks like this:

@table('table_name')
class MyDomainClass {
    @hashKey()
    id: string;

    @attribute()
    requiredProperty: string;

    @attribute()
    optionalProperty: string | null;
}

When I perform a query using @aws/dynamodb-data-mapper, the result is:

{
  "id": "foo",
  "requiredProperty": "bar",
  "optionalProperty": ""
}

I would expect optionalProperty to have value null. The corresponding document looks like:

{
  "id": { "N": 1},
  "requiredProperty": { "S": "Foo" },
  "optionalProperty": { "NULL": true }
}

startKey property of scan method automatically set the default value

Hi, I am tried to implement the paging the application and while doing i am setting the last accessed id in startkey property of scanOption

var option: ScanOptions = {
filter: notEqualExpression,
projection: projection
};
if (startKey) {
option.startKey = { id : startKey };
}
var query = mapper.scan(User, option);

while doing this I am getting this error "The provided starting key is invalid: The provided key element does not match the schema"

Here my model definition:
@table("Users")
export class User {
@hashkey() id: string;

@Attribute() email: string;

@Attribute() firstName: string;

@Attribute() lastName?: string;

@Attribute() photoUrl?: string;

@Attribute() heading?: string;

@Attribute() biography?: string;

@Attribute() phone?: string;

@Attribute({ defaultProvider: () => new Date() })
createdDate: Date;

@Attribute({ defaultProvider: () => true })
active: boolean;

@Attribute({ memberType: "String" })
deviceTokens?: JSONSet;

@Attribute() paypalEmail?: string;

@Attribute({ defaultProvider: () => false })
approved: boolean;

@Attribute({defaultProvider: ()=> 'basic'})
provider: string;
}

After the debugging this issue, i found that ExclusiveStartKey expression also contain the default value of attributes in which defaultProvider is provided.

Example Project

I like the library and the documentation is OK, but I am really missing examples and recipes.

Right now I am stuck on how to ensure that on a put that it fails if the ID already exists and allows me to retry with another randomly generated id. I assume that I should use some expression, but can't find an example or recipe.

I'd love to see recipes or sample project that show examples of how this code looks like, this would be very helpful.

A little question on update method.

The update method will overhide the data if I just send a partial of this data?
my data has a id and a name already.

mapper.update({
id: 'someid',
list: [a list]
})

If do this I will be erasing the name?

Typescript type-safe usage with a wrapper function

I'm trying to implement a wrapper nestJS service around the DataMapper. But, I'm not an expert with generics in Typescript and also unsure about AsyncIterator. This is what I have so far:

a DynamoDBDataMapperService:

import { Injectable } from '@nestjs/common';
import { DynamoDB } from 'aws-sdk';
import { DataMapper, ScanParameters } from '@aws/dynamodb-data-mapper';

@Injectable()
export class DynamoDBDataMapperService {
    private client: DynamoDB;
    private mapper: DataMapper;

    constructor() {
        this.client = new DynamoDB({
            region: 'us-west-2',
            endpoint: 'http://localhost:8000',
        });
        this.mapper = new DataMapper({ client: this.client });
    }

    public put<T>(item: T): Promise<T> {
        return this.mapper.put(item);
    }

    /**
     * Scans for an the passed item type
     * @param item generic dynamoDB data mapper annotation type
     * @returns AsyncIterator<T> Intended to be consumed with a `for await ... of` loop.
     */
    public get<T>(item: T): AsyncIterator<T> {
        const scanParameter: ScanParameters = { valueConstructor: item };
        return this.mapper.scan(scanParameter);
    }
}

and then a nestJS service that uses the wrapper:

import { Injectable } from '@nestjs/common';
import { CreateCatDto } from './dto/create-cat.dto';
import { Cat } from './schemas/cat.schema';
import { DynamoDBDataMapperService } from '../dynamodb-data-mapper/dynamodb-data-mapper.service';

@Injectable()
export class CatsService {
  constructor(private readonly mapper: DynamoDBDataMapperService) {}

  async create(createCatDto: CreateCatDto): Promise<Cat> {
    const createdCat = new Cat();
    createdCat.name = createCatDto.name;
    createdCat.age = createCatDto.age;
    createdCat.breed = createCatDto.breed;
    return this.mapper.put(createdCat);
  }

  async findAll(): Promise<Cat[]> {
    return await this.mapper.get(Cat);
  }
}

I'm running into 2 problems:

  1. When I try to set the return type of DynamoDBDataMapperService.get() to AsyncIterable<T> I get the error
[ts] Cannot find name 'AsyncIterator'.
  1. I could just be a noob at generics in Typescript, but I'm not sure how to pass the generic type along to DataMapper.scan()
[ts]
Type '{ valueConstructor: T; }' is not assignable to type 'ScanParameters<StringToAnyObjectMap>'.
  Type '{ valueConstructor: T; }' is not assignable to type 'CtorBearer<StringToAnyObjectMap>'.
    Types of property 'valueConstructor' are incompatible.
      Type 'T' is not assignable to type 'ZeroArgumentsConstructor<StringToAnyObjectMap>'.

reproduction case

https://github.com/CoreyCole/nest/tree/sample/dynamodb-data-mapper-annotations/sample/20-dynamodb-data-mapper-annotations

questions

I could simply solve my problem by instead exposing the mapper as a public property and then reference it directly in my CatsService, but I'd like to learn how to make this wrapper in a type-safe way. I'd like to submit this sample to nestJS for others trying to use dynamodb-data-mapper in a nestJS project using best practices as advised from you guys ๐Ÿ˜ƒ(why I'm not asking on stack overflow)

  1. How do I properly use AsyncIterator? It looks like it's been supported in Typescript since 2.3 and my example project is on 2.8
  2. How can I pass the generic type along in a ScanParameters object to DataMapper.scan()?
  3. Do you have any other suggestions to how I might improve my github example to better follow your suggested best-practices?

Collection or Custom, needing advice

Hey guys, needing some advice, @jeskew, maybe you know the answer...

I have Accounts, that have a phones attribute, that is a phone map.

{
 phones: {
  'phone1-with-uuid': {
       number: '99999999',
       areaCode: '00'
  }
 }
}

I was looking ate the collection and I can't use a memberType for it, the mapper will do a auto-marshaller, rigth?

There is a way to make a custom use the scheme? Or we extend the mapper to understand this use case?

I want to do this to have a unique identifier on phone.

Adjacency List Design Pattern

Per the DynamoDB docs one should only use a single table and use things like "Adjacency List Design Pattern" to implement this, see links below.

Ideally I would like to have to multiple domain objects that all point to the same table and handle the id automatically (e.g. order-15 for order id 15, and customer-123 for customer id 123).

How would I implement this with this library?

Refs:

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-adjacency-graphs.html#bp-adjacency-lists

Fields with references to data in S3

The AWS SDK for Java's DynamoDBMapper class includes a way to save large blobs to S3 and write only metadata to DynamoDB. This is done by declaring an attribute field with the type S3Link, which will eventually save a small JSON document to DynamoDB describing how to load the object from S3. This data mapper should implement something similar, and ideally it should be compatible with the SDK for Java's format.

Examples from the Java repo:

{"s3":{"bucket":"mybucket","key":"mykey","region":"us-west-2"}}
{"s3":{"bucket":"mybucket","key":"mykey","region":null}}

Since interacting with S3 is asynchronous in the JS SDK, this type will need special handling in the data mapper.

Unable to query using global secondary index

Hi,

I have created a table which has the below structure:

const params = {
    AttributeDefinitions: [
      {
        AttributeName: 'deploymentId',
        AttributeType: 'S',
      },
      {
        AttributeName: 'startedAt',
        AttributeType: 'N',
      },
      {
        AttributeName: 'botId',
        AttributeType: 'S',
      },
      {
        AttributeName: 'tenantId',
        AttributeType: 'S',
      },
    ],
    KeySchema: [
      {
        AttributeName: 'deploymentId',
        KeyType: 'HASH',
      },
      {
        AttributeName: 'startedAt',
        KeyType: 'RANGE',
      },
    ],
    GlobalSecondaryIndexes: [{
      IndexName: 'tenantId-botId-Index',
      KeySchema: [
        {
          AttributeName: 'tenantId',
          KeyType: 'HASH',
        },
        {
          AttributeName: 'botId',
          KeyType: 'RANGE',
        },
      ],
      Projection: {
        ProjectionType: 'ALL',
      },
      ProvisionedThroughput: {
        ReadCapacityUnits: 1,
        WriteCapacityUnits: 1,
      },
    }],
    ProvisionedThroughput: {
      ReadCapacityUnits: 1,
      WriteCapacityUnits: 1,
    },
    TableName: tableName,
    StreamSpecification: {
      StreamEnabled: false,
    },
  };

Now when I try to query using the data mapper using only the keys of the Indexed table like this:

const query = {
    indexName: 'tenantId-botId-Index',
    valueConstructor: ProvisionStatusRecord,
    keyCondition: {
      tenantId: tenantId,
      botId: botId,
    },
    scanIndexForward: true,
    limit: 1,
  };
const iterator = mapper.query(query);

it returns ValidationError as below:

ValidationException: Query condition missed key schema element: deploymentId

=============
It seems like it is searching for the base table Partition key which I guess is actually not required when querying on the Indexed table.

Please help

GSI pagination issues on version 0.4.1

Hey o/

First of all, we've been using dynamodb-data-mapper for a few months and it has been great, so thanks for the project!

One thing I've noticed is that GSI pagination is broken for me since 0.4.1.

Here's my table structure:

  • Table Name: offers
  • Hash: merchant_id
  • Range: object_id
  • GSI
    • IndexName: status_merchant_id
    • Hash: status
    • Range: merchant_id

The Exclusive Start Key of a GSI pagination usually looks like this:

{ 
  merchant_id: '1234,
  object_id: '456',
  status: 'INACTIVE' 
}

I'm getting this error message The provided starting key is invalid now when doing GSI queries.

I believe the error could be related to this PR:
https://github.com/awslabs/dynamodb-data-mapper-js/pull/40/files#diff-5308a7ee95cbc7142d35cd94cd31c718

I'm not familiar with the inner workings of the project, but it seems that only the table keys will be serialized, but not the keys from my GSI. Because status was not serialized, that error is happening.

Does that make sense?

Working example

I am having trouble using this with TS.
Can we provide a working example with proper tsconfig.json?
I have experimentalDecorators and emitDecoratorMetadata on, and imported "reflect-metadata", and I still get this:

Error: Properties of type Any may not be used as index or table keys. If you are relying on automatic type detection and have encountered this error, please ensure that the 'emitDecoratorMetadata' TypeScript compiler option is enabled. Please see https://www.typescriptlang.org/docs/handbook/decorators.html#metadata for more information on this compiler option.

UpdateExpression with DataMapper.update

Is it possible to use an UpdateExpression with the DataMapper.update method?

I think the docs around advanced usage could use a little more detail and examples :) I am happy to pull request some of this, but I haven't been able to get much of it working myself!

Q: Annotated Model - Import in node.js function - Annotation lost error (No object property was found)

Hi

I am really digging the concept and would like to apply the given SDK here. The issue I am facing is the following:

I have a node application and would like to create model classes shared for different modules. These models should define their own DynamoDbTable and DynamoDbSchema. Everything is JavaScript es2015.

Now, if I define the classes and export them I can reuse them but the symbols are not created/available. I would have to reset the configuration everywhere again where I would like to use the model class together with the datamapper to DynamoDB, otherwise it ends in

Error: The provided item did not adhere to the DynamoDbDocument protocol. No object property was found at the "DynamoDbSchema" symbol

I tried with Symbols in class, Object.defineProperties, as in the examples, but reusing the object only seems to grab an "interface" of it, without any content, even if I define e.g.
[DynamoDbTable]() { return 'posts'; }
in the class.

Is there any way to modularize this somehow without TypeScript or an upgrade NodeJS version? Only if I Object.defineProperties always again where I am using the model it does work.

Thanks in advance

mapper.put not return objectId

Hello, I'm using with serverless and when I'm trying to mapper.put () this is not returning object. See my example thanks.

export async function createArtist(args) {
const artist = new Artist();
artist.firstName = args.first_name;
artist.lastName = args.last_name;
return await new Promise((resolve, reject) =>
mapper.put({item: artist})
// tslint:disable-next-line:arrow-parens
.then(data => resolve(data))// data is undefined
// tslint:disable-next-line:arrow-parens
.catch(err => reject(err)),
);
}

@table("artists")
class Artist {

@autoGeneratedHashKey()
@attribute({keyType: "HASH"})
public id: string;

@attribute()
public firstName: string;

@attribute()
public lastName: string;

}

Getters and setters on domain objects

What are the best practices for using getters and setters on domain objects?

For example I would like to hash the password field every time it is set.

class User {
  set password(password) {
    const salt = bcrypt.genSaltSync(10);
    this.password = bcrypt.hashSync(password, salt);
  }
}

However this obviously will result in an infinite loop causing a stack overflow.

I was hoping to be able to define the field mapping in the schema to overcome this but I don't see documentation relating to field mapping, for example:

Object.defineProperties(User.prototype, {
  [DynamoDbSchema]: {
    value: {
      password: {
        field: '_password',
      },
  },
});

Types not autodetected with serverless-plugin-typescript

I noticed that when using https://github.com/graphcool/serverless-plugin-typescript to deploy TypeScript based Lambda applications, this doesn't work:

@table(process.env.CONTACTS_TABLE!)
class Contact {
  @hashKey()
  userId: string
  @rangeKey()
  contactId: string
}

The type of all attributes becomes 'Any', which causes e.g. DataMapper.get() to throw a ValidationException: The provided key element does not match the schema, because it can't figure out how to marshal the key.

My workaround is to manually specify the types:

@table(process.env.CONTACTS_TABLE!)
class Contact {
  @hashKey({ type: 'String' })
  userId: string
  @rangeKey({ type: 'String' })
  contactId: string
}

Is there some option serverless-plugin-typescript could use to keep the type metadata around when building the Lambda code?

Only put when item doesn't already exist

Apologies for raising an issue again. I've been struggling for longer than I'd like to admit on this.

When doing a put I want to only succeed if an entry with the id doesn't already exist. Reading the dynamodb docs it's done using attribute_not_exists(id).

I'm a little stuck in determining how to formulate that condition using dynamodb-expressions.

Here's my attempt.

const release = new Release()
release.id = 'foo'
const condition = new FunctionExpression(
    'attribute_not_exists', 
    new AttributePath([{type: 'AttributeName', name: 'id'}]),
)
await mapper.put(release, { condition }) 

This gives me a ValidationException: ExpressionAttributeValues must not be empty error but I can't for the life of me work out how to solve this (maybe I need new AttributeValue({ S: release.id }) somewhere?), only that I'm guessing it's quite simple!

Thanks again for the work on this lib.

Is there a way to set onInvalid throw at the document level?

Is it possible to throw an error when trying to save an object with not defined attributes?

For example: if I define a schema only with firstName, then I try to save lastName, is there a way to cause it to raise an error? Currently, it is silently ignoring the any not defined value, but I would rather receive an error.

I tried (random attempt) to use onInvalid: 'throw', but it didn't work.

Object.defineProperties(Model.prototype, {
  [DynamoDbTable]: {
    value: 'table',
  },
  [DynamoDbSchema]: {
    onInvalid: 'throw',
    value: {
      hash: { type: 'String', keyType: 'Hash' },
      range: { type: 'String', keyType: 'Range' },
      firstName: { type: 'String' },
      data: {
        type: 'Document',
        members: dataSchema,
        onInvalid: 'throw'
      }
    }
  }
})

Parallel scan

The data mapper's scan operation currently only supports sequential scan operations. It should be updated to allow parallel scans to be executed (while still maintaining the same asynchronous iterator-based interface).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.