batch write possibilities? about dynogels HOT 8 OPEN

@iDVB I agree with your suggested structure: a batchWrite method that handles both put and delete functionality. create and destroy call batchWrite when invoked with an array.

from dynogels.

@dangerginger & @cdhowie

Is there anything new on this topic, or do you know who is building it atm? 👍

from dynogels.

You can pass an array to the dynogels create function.

Here's our usage (with https://github.com/andrewoh531/dynogels-promisified and https://github.com/lodash/lodash/wiki/FP-Guide):

function bulkCreate(orders) {
  return orderTable
    .createAsync(orders)
    .then(map('attrs'));
}

function create(order) {
  return orderTable
    .createAsync(order)
    .then(get('attrs'));
}

I'm assuming you want to do the bulk multi type write? E.g. some deletes, some edits and some creates all in one request?

from dynogels.

+1

from dynogels.

I would be interested to see this as well. I will look into it.

from dynogels.

The way I read this is that if you supply an array to create() then dynogels will send one query for every item in the array. A batch create would send only a single query to dynamodb with an array of items. Can you clarify this one @stevenao or @M1chaelTran?

from dynogels.

As per #123, the API call being discussed allows putting multiple items in a single API call. I believe we can simply reimplement the array case of model.create() to use this API call and gain performance without losing functionality.

Note that this API call does have a limit on the amount of data that can be sent in one call (25MB) so we would need some logic to potentially split up items into multiple batches.

from dynogels.

@cdhowie you may run into nomenclature challenges then. batchWrite actually supports put AND delete.

The BatchWriteItem operation puts or deletes multiple items in one or more tables. A single call to BatchWriteItem can write up to 16 MB of data, which can comprise as many as 25 put or delete requests. Individual items to be written can be as large as 400 KB.
http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html

Where as dynogels model.create(), I believe, only handles creating.

I would suggest reimplementing the array case of model.create() also inside of model.destroy() but it sounds like @clarkie is right and that someone may want to do a mixed request which would feel weird making it from .create() or .destroy().

Perhaps creating a model.batchWrite as well as reimplementing array model.create() and doing the same for model.destroy?

from dynogels.