pageuppeople-opensource / dynamo-incremental-restore Goto Github PK
View Code? Open in Web Editor NEWAbility to restore S3 incremental backups to DynamoDb to any point in time.
License: MIT License
Ability to restore S3 incremental backups to DynamoDb to any point in time.
License: MIT License
The object versions retrieved from s3.listObjectVersions (buildS3VersionList.js) will contain a directory key as well. For example, if you have put incremental object records in:
backup/
then a object version of backup directory itself will also be pass to s3ToDynamo.js:
Therefore, the following exception will be throw:
exception in /Users/han/repo/dynamo-incremental-restore/lib/s3ToDynamo.js:40
SyntaxError: Unexpected end of input
38 console.timeEnd('RFS3' + version.Key);
39 try {
>40 deferred.resolve([version, JSON.parse(data)]);
41 } catch (e) {
42 deferred.reject('Failed to retrieve file from S3 - Params: ' + JSON.stringify(s3Params));
Because the data will be empty so it will be an invalid JSON file. Just wondering is this issue because my prefix setting? I have the following map pass to logbuildList function.
{
Prefix: "backup/registrations/",
//Adjust the InstID as needed. OR you can not provide an Inst ID and you get all the forms.
// e.g. Formsmith/Formsmith.Green.FormData/543
DestinationTableName: "registrations" ,
//The table to backup. 1 table at a time e.g. Formsmith.Green.FormData
Bucket: "blanco-han-test",
//The bucket where the data is stored. e.g. pageup-dynamo-backup
restoreToPointInTime: new Date('2017-01-12T18:34:34.128Z'),
//the last point in time you want to update a version
//restoreToPointInTime: new Date('2016-11-14T23:50:32.000Z'),
}
If so I am curious how do your guys set up the incremental backup so that the directory will never be included.
BTW, I also made some fix for the signature version 4 signing which will follow later.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.