Is backward pagination supported? about dynamodb-onetable HOT 17 CLOSED

Sure!

I'll build a working example using hidden=true and get back to you.

I think I'll have some time to spend on this during this week. I know this pattern works as I have implemented it many times using aws-sdk directly.

from dynamodb-onetable.

I've checked in a prototype where items.prev is set to the pk/sk of items[0].

To page backwards, set reverse == true and params.start == items.prev

from dynamodb-onetable.

I've checked in a prototype and updated the test/pagination test case to demonstrate similar to your sample.

Could you please look it over and see if I've got it right for your needs?

from dynamodb-onetable.

I have updated my tests to use next/prev and everything seems to work fine 🥳

I totally understand your point about Param.fields, and you are right, not all use cases require pagination. Explicitly adding key attributes to Params.fields looks a good tradeoff.

Regarding documentation, I would clarify these points:

The params.fields may be set to a list of properties to return. This defines the ProjectionExpression.

The params.fields may be set to a list of properties to return. This defines the ProjectionExpression. Be aware that key attributes must be included in order to enable pagination.

If params.next is set to a map that contains the primary hash and sort key values for an existing item, the query will commence at that item.

I would extend that phrase or add a new paragraph for params.prev. Furthermore, I assume they can't be used at the same time, so I would clarify that too:

If params.next or params.prev is set to a map that contains the primary hash and sort key values for an existing item, the query will commence at that item. These two properties are mutually exclusive, both of them can't be set at the same time.

from dynamodb-onetable.

BTW I would like to thank you, not only for the library which is pretty awesome, but also for your dedication and quick replies/implementation 😃

Thank you!

from dynamodb-onetable.

Thanks very much for your feedback and suggesions. Let me dig into this as this may be possible already. start indicates the key to being the query. reverse already supported sets the ScanIndexForward. Are you saying this does not work, or are you saying you'd like to see reverse implemented?

Can I please ask why would the design need a first vs start if reverse is set to true which would indicated that the pagination should happen in reverse? Is this just a bug or is the design lacking?

from dynamodb-onetable.

I believe the design is missing a feature.

Maybe I should have given an example.. here it is:

Assume you have a partition with 100 items. SK is composed somehow that the partition is sorted in ascending order, lets say:

item 1
item 2
item 3
.
.
.
item 100

It's clear to me that you can use start property to navigate forward -->

1. [item 1, item 2, ..., item 10]
2. [item 11, item 12, ..., item 20]
3. [item 21, item 22, ..., item 30]
4. and so forth.

Now imaging you are in page n and you want to get page n-1. You can't use start property here as it stores the key attributes of the last item in page n. You would need the first item's key attributes as LastEvaluatedKey, set ScanIndexForward to false, and finally reverse the returned list (otherwise results would be reversed --> [item n-1, item n-2, ..., item n-limit]).

ScanIndexForward: false can be easily set using dynamodb-onetable's reverse property. But I haven't seen any way to get first item's key attributes as easily as I can get last item's key attributes through the start property.

Similarly, imagine now the partition is sorted in descendent order (for instance using dates and you want more recent first). The partition would look like:

item 100
.
.
.
item 3
item 2
item 1

Now, iterate through this collection you would set reverse: true by default. That would give you pages like:

1. [item 1, item 2, ..., item 10]
2. [item 11, item 12, ..., item 20]
3. [item 21, item 22, ..., item 30]
4. and so forth.

Again, if you wanted page n-1, you would need page's first item's key properties, set reverse: false, and finally reverse the returned list.

So, summarizing, start property stores a page's last item's key properties which easy forward pagination. For backward pagination, the starting point isn't a page's last item's key properties, but a page's first item's key properties.

Am I missing anything? Can this be done and I haven't seen it?

from dynamodb-onetable.

Thanks for the extra detail -- appreciated, let me play around a bit and see what is possible.

from dynamodb-onetable.

Just completed a test.

See the sample below. We create 1000 users with the schema below. We simply set reverse to true and use start to mean the start point of the reverse scan.

export default {
    indexes: {
        primary: { hash: 'pk', sort: 'sk' },
    },
    models: {
        User: {
            pk:         { type: String, value: 'user#' },
            sk:         { type: String, value: 'user#${name}' },
            id:         { type: String, uuid: true },
            name:       { type: String, required: true },
        }
    }
}

You can scan backwards using start.

        await User.create({name: `user-${zpad(i, 4)}`})
    }
    let items = await User.scan()

    //  Scan forwards
    let start = null
    do {
        let items = await User.find({}, {limit: 100, start, reverse: true})
        print(`GOT ${items.length}`)
        start = items.start
    } while (start)

from dynamodb-onetable.

Thanks for the example @mobsense !

As far as I can tell, that will traverse the table in backward order. However that's not backward pagination. The example above always asks for the next page. Backward pagination means the ability to ask for the previous page, i.e. once you get page n, being able to get page n-1.

In your example, once you get page n, you always asks for page page n+1

from dynamodb-onetable.

I don't think so. Each find gets the page before the current page. So you are stepping backwards in pages. The items are in reverse order, but you can do an items.sort() to get each page in forward order if you need.

When you use a start that will be the starting key, and reverse will then get the items before that. That is the best dynamo can do I think.

Or am I missing something?

from dynamodb-onetable.

There are two different things here. On the one hand, you have the order: forward or backward. You can set that order using reverse flag (which directly maps to dynamo's ScanIndexForward property). This gives you the ability to traverse a collection in one direction or the other.

On the other hand, you have pagination, which can be forward (give me the next page), or backward (give me the previous page).

To ask for the next page, you set ExclusiveStartKey to the last item within the current page and you keep ScanIndexForward as it was. That means keep the same order and return items from the last one I know.

To ask for the previous page, you set ExclusiveStartKey to the first item within the current page, you flip ScanIndexForward and call response.Items.reverse(). That means traverse the collection in the opposite order as it was and return items from the first one I know. That will return the previous page in the wrong order, that's why you need the extra response.Items.reverse().

This pattern is specially useful when you feed an API. Your clients might not cache previous pages or content within the collection might be added to a previous page (live feed for instance). So, you must provide clients with a way to get previous pages.

For instance:

1. client app requests a page of users (GET /users). The backend does:

items = await User.find({}, {limit: 100});
return pageResponse(items);

which returns the list of items plus cursors to get the prev and next page. Something like:

{
   items,
   prev: "encoded string", // encodes the key attributes of the first item plus a flag, isBackward=true
   next: "encoded string", // encodes the key attributes of the last item plus isBackward=false
}

2. client app now wants to know next or previous page, so it calls GET /users?cursor="prev or next cursor". The backend does:

{key, isBackward} = decryptCursor(cursor);
items = await User.find({}, {limit: 100, start: key, reverse: isBackward});
if (isBackward) {
   return pageResponse(items.reverse());
}

return pageResponse(items);

With dynamodb-onetable is easy to compose the next cursor, I can just use items.start (which is indeed the key attributes of item[item.length - 1]), however I can't compose the prev cursor easily. There is no shortcut to get the key attributes of items[0].

Hopefully this clarifies the use case I'm missing in the library.

from dynamodb-onetable.

Thank you for the extra detail.

Can you try Params.hidden == true. Then you will get the PK/SK values for all items. Then construct the start value using the items[0] keys.

If this works, we could then do that internally and return an items.prev set to the primary key of the first item.

Does that make sense?

from dynamodb-onetable.

Here is a working example of what I meant by forward and backward paging:

https://github.com/cjuega/dynamodb-onetable-backward-pagination-example

It basically does what I described in my previous comment.

TLTR:

prevPage = (
    await User.find(
        {},
        {
            limit: 20,
            // It would be nice to have a shortcut in the same fashion as when paginating forward
            start: { pk: nextPage[0].pk, sk: nextPage[0].sk },
        }
    )
).reverse();

from dynamodb-onetable.

Thank you for the working example, that helps.

A question about the high level API usage:

It looks like this:

let page = await User.find({}, {limit})
let nextPage = await User.find({}, {limit, start: page.start})
let prevPage = await User.find({}, {limit, start: page.prev, reverse: true}).reverse()

Perhaps we could dry it up with a Params.prev which would imply the double reverse.

let page = await User.find({}, {limit})
let nextPage = await User.find({}, {limit, start: page.start})
let prevPage = await User.find({}, {limit, prev: page.prev})

And then, since Params.next is deprecated, we could alias Params.start to Params.next and then we have a nice symmetrical next/prev cursor?

One other issue:

If doing a Params.fields which defines a ProjectionExpression to return a subset of fields, you may not have access to the keys in the returned items. next/prev can only work if the result set includes both keys. If you use Params.fields, you must include the keys or set hidden: false.

from dynamodb-onetable.

Perhaps we could dry it up with a Params.prev which would imply the double reverse.

Sure! That would be helpful. I can't think of a use case in which you wanted the collection without reversing. And if it exists, we can always reverse it manually.

Are you also saying to manage reverse: true implicitly? That can be done, but it's tricky. We must know in advance the direction in which we are traversing the partition (which I guess we know as clients must set reverse property anyway):

• If we are traversing the partition forward, i.e. reverse: false when fetching the current page, then, to get prev page reverse must be true.
• If we are traversing the partition backward, i.e. reverse: true when fetching the current page, then, to get prev page reverse must be false.

So, we have to negate reverse value to get prev page. If you want dynamodb-onetable to do it automatically, then whenever Params.prev is set, the library must do something like reverse = !Params.reverse.

That would be helpful because it's easy to mess things up if the library delegates that responsibility to clients.

The high level API would be:

// when traversing the partition forward
let page = await User.find({}, {limit})
let nextPage = await User.find({}, {limit, start: page.start})
let prevPage = await User.find({}, {limit, prev: page.prev}) // this will set Params.reverse to true and it will also reverse the resulting list

// when traversing the partition backward
let page = await User.find({}, {limit, reverse: true})
let nextPage = await User.find({}, {limit, start: page.start, reverse: true})
let prevPage = await User.find({}, {limit, prev: page.prev, reverse: true}) // this will set Params.reverse to false and it will also reverse the resulting list

Would that make sense?

And then, since Params.next is deprecated, we could alias Params.start to Params.next and then we have a nice symmetrical next/prev cursor?

Yes, that makes much more sense as next/prev are more semantic.

If doing a Params.fields which defines a ProjectionExpression to return a subset of fields, you may not have access to the keys in the returned items. next/prev can only work if the result set includes both keys. If you use Params.fields, you must include the keys or set hidden: false.

This issue happens also with the current implementation, doesn't it? I mean, if somebody uses Params.fields, page.start won't work unless fields include key attributes, right? Would it make sense to you to always include key attributes regardingless Params.field? That would always enable pagination without impacting too much performance.

from dynamodb-onetable.

I think the code for handling params.reverse and params.prev is an XOR.

args.ScanIndexForward = (params.reverse != null ^ params.prev != null) ? false : true

I'm not sure about the Parmas.fields yet and want to think a bit more about it. I don't want to ignore the user Params.fields and add extra fields under the hood. Users typically only use that in really performance sensitive cases where minimizing the I/O is worth it. I can definitely warn about it in the doc. It is hard to detect as every scan/find is meant to return the next/prev keys.

from dynamodb-onetable.