TypeScript support about dynamodb-onetable HOT 14 CLOSED

You can do quite a magic things with TypeScript, and having the types generated in runtime would be much better than executing a separate command to generate them. Someone will forget to run a script and you have build passing, but with inconsistent models.

Some PoC on how to infer the model type in runtime:

const model = {
    Account: {
        name: {type: String},
        age: {type: Number},
    },
};

interface Field {
    type: StringConstructor | NumberConstructor;
}

type FieldType<T extends Field> =
    T['type'] extends StringConstructor ? string
        : T['type'] extends NumberConstructor ? number
        : never;

type Model<T extends Record<string, Field>> = {
    [P in keyof T]: FieldType<T[P]>;
}

const account: Model<typeof model.Account> = {
    name: 'aa',
    age: 3,
};

The same approach can be used to make fields optional or not, etc. And of course, the end goal would be not to use typeof model.Account by the user, but require proper types in your methods like find().

from dynamodb-onetable.

I also think TypeScript would be a killer feature for dynamodb-onetable. 🔥

And I agree with @m-radzikowski that it's much much better to have the "runtime" types. 👍

from dynamodb-onetable.

We've pushed a prototype 1.2.0 which has an implementation for folks to test.

Previously the *.d.ts files were generated by typescript. Now, the declaration files are hand-crafted for Table and Model.

There is some doc in the README and a stub sample under doc.

Features

• Model and Table APIs are typed.
• Schemas dynamically converted to types without any manual step.
• Schemas indexes and models are fully typed via the Table and Model constructor APIs.
• Entity types created from schemas with access to properties fully typed.
• The Model level API is fully typed -- this is the preferred access pattern.
• The Table get/update/find APIs are not generic and not typed. You need to cast the return.

When using TypeScript, the core change is to create an Entity type and then use the generic Model<> constructor. e.g.

import {Entity, Model, Table} from 'dynamodb-onetable'
type Account = Entity<typeof schema.models.Account>
let AccountModel: Model<Account> = table.getModel('Account').     // or
let AccountModel = new Model<Account>(table, 'Account')

let account = await AccountModel.get({id: 'xxxx'})

I've not yet worked out a simpler way to say "typeof schema.models.Account", but that is done once per compilation unit.
Folks may have different preferences for naming vs Account, AccountModel and account. i.e. Entity, Model and instance.

Examples:

import {Entity, Model, Table} from 'dynamodb-onetable'
import DynamoDB from 'aws-sdk/clients/dynamodb'
import schema from './schema.js'

const client = new DynamoDB.DocumentClient()

const table = new Table({ name: 'MyTable', client, schema })

//.  Create an entity type
type Account = Entity<typeof schema.models.Account>

//.   Create an access model
let AccountModel = new Model<Account>(table, 'Account')

//.   Fully typed access for return entity and for get parameters and options
let account = await AccountModel.get({id: '1234'}, {log: true})

//.   Stand alone schema
let BlogSchema = {
    pk:        { type: String, value: 'blog:${email}' },

    email:     { type: String, required: true },
    message:{ type: String, required: true },
    date:      { type: Date, required: true },
}

//.   Add the schema
table.addModel('Blog', BlogSchema)

//.   Get an entity type
type Blog = Entity<typeof BlogSchema>

//.   Retrieve a model from the table. Cast required.
let BlogModel: Model<Blog> = table.getModel('Blog')

//.   Fully typed access
let blog = await BlogModel.get({email: '[email protected]'})
blog.email = '[email protected]'.   // OK
blog.unknown = 42. // Fail

//  Untyped parameters and result
blog = await table.get('Blog', {name: '[email protected]'})

from dynamodb-onetable.

The library can already be used by Typescript and we supply d.ts files and interfaces (generated).

I'm gathering requirements for improved TypeScript support. This means different things to different people. Can you list what your looking for.

from dynamodb-onetable.

Sure. I guess the schema param for the table should extend some interface, and then the Table should take the schema type to type check further usage. Like so:

const MySchema = {
    indexes: {
        primary: { hash: 'pk', sort: 'sk' }
        gs1:     { hash: 'gs1pk', sort: 'gs1sk' }
    },
    models: {
        Account: {
            pk:          { value: 'account:${name}' },
            sk:          { value: 'account:' },
            id:          { type: String, uuid: true, validate: /^[0-9A-F]{32}$/i, },
            name:        { type: String, required: true, }
            status:      { type: String, default: 'active' },
            zip:         { type: String },
        },
};

const table = new Table<MySchema>({ // the generic type can be inferred from the param, I put it here for readability
    client: client,
    name: 'MyTable',
    schema: MySchema, // schema param type must be of some interface, let's say built-in OneTableSchema
});

where OneTableSchema would be something like this:

interface OneTableSchema {
  indexes: Record<string, { hash: string, sort?: string }>;
  models: Record<string, OneTableModel>
}

type OneTableModel = Record<string, OneTableField>;

interface OneTableField {
  value: string;
  uuid?: boolean;
  ...
}

Then all functions like find({name: '...'}) should expect the object passed as param to match the defined schema (so I cannot pass object with firstname as this does not exist in the schema), and returned response should be a typed object (type constructed from the OneTableModel definition in this case, not easy but doable).

All of the above is just a rough idea of course. Feel free to ask if something is not clear!

from dynamodb-onetable.

Thank you.

Can the type definitions be generated dynamically, or is that a build time too to generate type files?

from dynamodb-onetable.

I'm not sure if I understand your question.

From the usage point of view, the type definitions are provided with the library. The types for Model functions, like find(), since the Model would have a generic type, would be interfered from the generic type. So you could say it's dynamic, I guess.

From the library point of view, so far I always generated types from the TypeScript when building a package. But for pure JS projects you can just add typing files and bundle them with the library.

from dynamodb-onetable.

Thanks.

The core API looks pretty easy. We already supply generated d.ts files that we can tighten up by hand. Alternatively, convert the four *.js source files to .ts. The total line count is ~1.8k lines, so should not be too bad.

But the bigger issue is the schema and entity types in the use schema. How do we generate the types for that? I presume we need the onetable CLI to generate them from the Schema. Or is there a better way?

Consider:

const MySchema = {

    models: {
        Account: {
            pk:          { value: 'account:${name}' },
            sk:          { value: 'account:' },
            id:          { type: String, uuid: true, validate: /^[0-9A-F]{32}$/i, },
            name:        { type: String, required: true, }
            status:      { type: String, default: 'active' },
            zip:         { type: String },
        },

You want a type Account that specifies the attributes: id, name, status, zip, ... You want them to be required or optional as specified in the schema.

So my question is how is this best achieved?

Sorry, I'm not deep on Typescript, but we're really committed to implementing this capability.

from dynamodb-onetable.

I've played around using the OneTable CLI to generate type definitions for the schema entities from the schema source.

Generates a models.d.ts that can then be included. Looks like:

export type Account = {
    email: string;
    id?: string;
    name: string;
}

export type User = {
...
}

Consume via

import {Account, User, Product} from './models'

then would construct like:

import {Model, Table} from 'dynamodb-onetable'
import DynamoDB from 'aws-sdk/clients/dynamodb'
import MySchema from './schema'

const table = new Table<MySchema>({
    client: new DynamoDB.DocumentClient(),
    schema: MySchema,
}))

let user = new Model<User>(table, ...)
let account = new Model<Account>(table, ...)

await user.create({
    // These attributes will be validated against the User in the generated models.d.ts
    name: 'user',
    email: '[email protected]',
})

Plenty to DRY up, but the prototyping looks promising. The generated JS looks okay so far.

This approach should be able to hit the 2 goals:

• Have type definitions for the schema and OneTable APIs parameters and return values
• Generate type definitions from the Schema for entities and attributes

Driving it all from the schema works really well.

from dynamodb-onetable.

Thanks @m-radzikowski, I'll work through your POC today, but a quick question.

Where does the Account type come from: from the user code, or from the schema?

const model = {
    Account: {
        name: {type: String},
        age: {type: Number},
    },
};

These MUST match the schema exactly.

Schemas have many attributes, other than type:

            crypt           Boolean
            enum            Array of values
            foreign         model:key-attribute
            hidden          Boolean
            map             String
            nulls           Boolean
            required        Boolean
            size            Number (not implemented)
            type            String, Boolean, Number, Date, 'set', Buffer, Binary, Set, Object, Array
            unique          Boolean
            validate        RegExp or "/regexp/qualifier"
            value           String template, function, array

So I guess you could have the users code their own models AND create the schema and then they must keep both in sync.

Alternatively, and optionally, the user could generate these via the OneTable CLI:

onetable generate types

Am I missing something?

from dynamodb-onetable.

No, I assume the whole idea is to have only ONE definition of the schema, to prevent them from going out of sync.

In the code I posted above the model was just a simplification. It was a section of the "schema". So in reality it would be your standard:

const MySchema = {
    indexes: {
        primary: { hash: 'pk', sort: 'sk' }
        gs1:     { hash: 'gs1pk', sort: 'gs1sk' }
    },
    models: {
        Account: {
            pk:          { value: 'account:${name}' },
            sk:          { value: 'account:' },
            id:          { type: String, uuid: true, validate: /^[0-9A-F]{32}$/i, },
            name:        { type: String, required: true, }
            status:      { type: String, default: 'active' },
            zip:         { type: String },
        }
    }
}

and types can be extracted from MySchema.models.Account etc.

from dynamodb-onetable.

Great, thanks I now get it.

Was your POC a working POC or just a hint of a direction to go?

Also, would you like to work on this yourself? Always keen to have collaborators.

from dynamodb-onetable.

It's some point of start to create full support for types.

Sorry, I can't commit to work on this, I am not able to spend time on it right now.

from dynamodb-onetable.

Thanks @m-radzikowski for your guidance. Really appreciated. Your design looks to be effective.

After some prototyping I get the following. Note this is incomplete on many fronts. Any ideas to DRY it up are appreciated.

I've renamed some of your types and applied to the Model class and get() API.

User code

import {Model} from 'dynamodb-onetable'

const schema = {
    models: {
        Account: {
            id: {type: String},
            name: {type: String, required: true},
            age: {type: Number, default: 23},
        },
    }
}
type Account = OneEntity<typeof schema.models.Account>

let account: Account = {
    id: '1234',
    name: 'Peter Smith',
    age: 33,
}

let AccountModel = new Model<Account>(table, 'Account');

account = AccountModel.get({id: '1234'})

The declarations to achieve this would be in the imported library via Model.d.ts:

interface OneTypes {
    type: StringConstructor | NumberConstructor | BooleanConstructor | ObjectConstructor | ArrayConstructor | DateConstructor;
}

type OneField<T extends OneTypes> =
      T['type'] extends StringConstructor ? string
    : T['type'] extends NumberConstructor ? number
    : T['type'] extends BooleanConstructor ? boolean
    : T['type'] extends ObjectConstructor ? object
    : T['type'] extends DateConstructor ? Date
    : T['type'] extends ArrayConstructor ? []
    : never;

type OneModel = Record<string, OneTypes>;

type OneEntity<T extends OneModel> = {
    [P in keyof T]: OneField<T[P]>;
}

type OneParams = {
    add?: object,
    batch?: object,
    capacity?: string,
    consistent?: boolean,
    context?: object,
    delete?: object,
    execute?: boolean,
    exists?: boolean,
    hidden?: boolean,
    index?: string,
    limit?: number,
    log?: boolean,
    many?: boolean,
    metrics?: boolean,
    parse?: boolean,
    postFormat?: () => {},
    preFormat?: () => {},
    remove?: string[],
    return?: string,
    reverse?: boolean,
    start?: boolean,
    throw?: boolean,
    transaction?: object,
    type?: string,
    updateIndexes?: boolean,
    where?: string,
}

type OneOptions = {
    indexes?: object,
    fields?: object,
    timestamps?: boolean;
}

type OneProperties = {
    [key: string]: any;
}

declare class Model<T> {
    constructor(table: string, name: string, options?: OneOptions);
    get(properties?: OneProperties, params?: OneParams): T;
    ....
}

Issues and Notes

• The properties for get() and other API methods can't really be typed effectively as the properties are blended with the table context. i.e. you may provide any key, non-key properties or none at all to the API.
• Still need to cover Binary and Set data types in OneTypes
• I've used "One" prefixes to avoid name clashes.
• Need to give you @m-radzikowski credit for the POC and concept. Thank you!

Thoughts?

from dynamodb-onetable.