Dynamo DB Pagination

In this article, let us look at pagination and how pagination is done with DynamoDB. Pagination is essential when we have a lot of items to display on the UI and we don’t want the user to wait till all the items are loaded. 

What is DynamoDB?

DynamoDB is a fully managed NoSQL database service from Amazon that provides fast and predictable performance with seamless scalability. See more over here

Prerequisites:

Basic understanding of DynamoDB tables, keys, indexes, and operations.

What is Pagination?

Pagination is splitting a database output into manageable chunks or pages. To do this:

1. Decide on the maximum number of database rows that can be included in each page. You may hard code this value, or you define it in a variable so that the value may be changed at runtime.
2. Inform the user that other ‘pages’ are available and provide a mechanism whereby the user is able to select a different ‘page’ of details. I currently use a set of hyperlinks in a separate pagination area which looks like:

Pagination Area

What pagination support does DynamoDB provide?

Amazon DynamoDB documentation says that DynamoDB paginates the results from scan/query operations. With pagination, the scan results are divided into “pages” of data that are 1 MB in size (or less). An application processes the first page of results, then the second page, and so on.

A single scan will only return a result set that fits within the 1 MB size limit. To determine whether there are more results, and to retrieve them one page at a time, your application should do the following:

1. Examine the Scan result:
   • If the result contains a LastEvaluatedKey element, proceed to step 2.
   • If there is no LastEvaluatedKey in the results, then you are done. There are no more items to be retrieved.
2. Construct a new Scan request, with the same parameters as the previous one—but this time, take the LastEvaluatedKey value from step 1 and use it as the ExclusiveStartKey parameter in the new Scan request.
3. Run the new Scan request.
4. Go to step 1.

In other words, the LastEvaluatedKey from a Scan response should be used as the ExclusiveStartKey for the next Scan request.

If there is not a LastEvaluatedKey element in a Scan response, then you have retrieved the final page of results. (The absence of LastEvaluatedKey is the only way to know that you have reached the end of the result set.)

Scan Query response contains the following elements:

• ScannedCount— the number of items that match the key condition expression, before a filter expression (if present) is applied.
• Count— the number of items that remain, after a filter expression (if present) is applied.

Note:

If you do not use a filter expression, then ScannedCount and Count will have the same value.

To find more about DynamoDB pagination click here

What DynamoDB pagination means?

DynamoDB does not provide the support to get the total items count for a scan/query operation. It provides the fetched items count for a single scan/query. The scan/query operation can fetch a maximum of 1MB data at a time. With DynamoDB, data for a particular page like page 4, 8 cannot be directly fetched as LastEvaluatedKey for that page is not known. LastEvaluatedKey is known only in sequence.

So, DynamoDB cannot help us achieve page sequencing, landing to a random page, jumping to the last page etc. What can be done in pagination with DynamoDB is, having previous or next or load more button.

How pagination is done?

• On the webpage where pagination is supported, the browser sends the LastEvaluatedKeyto the server.
• To access the first page, browser sends the LastEvaluatedKeyas undefined. With this undefined key, server sends the first page data and also sends the LastEvaluatedKey for the next page.
• To access the next page, the browser sends back the LastEvaluatedKeyand the same procedure continues.

Enough of theory, let’s implement it!!

Here, I have implemented blog search by title functionality with the following:

• A blog table with bid (blog ID) as the primary key
• BlogSearch is a GSI on the blog table. The GSI is added to have the search by title functionality.
• For the BlogSearch GSI, primary key is a fixedID string that is same for all the items
• title_bid is the sort key

NOTE: In DynamoDB, LastEvaluatedKey for table is an object that contains table keys as properties. LastEvaluatedKey for an index is an object that contains tables and index keys as properties.

https://gist.github.com/mayank-kansal15/9dc98676b0e28d544816a4a4ad434eae

DynamoDB Bug: In dynamoDB pagination, the lastEvaluatedKey should be undefined when we access the last page as there are no more items. But when we access the last page and itemsPerPage is equal to the items left in DB, then instead of giving LastEvaluatedKey as undefined, DynamoDB give it as an object.

Conclusion:

Pagination is very different from SQL databases, and ensuring that you receive complete data is important.