martin-opensky / kwil-extension-db-sync Goto Github PK

This tool includes 2 repositories: one is for an extension and the other is for a gRPC server that runs as a container alongside the sync extension. This tool saves database actions to Arweave (through Bundlr) along with specific tags enabling actions to be easily traced and synced with other Kwil Providers running the service.

gRPC Server: https://github.com/martin-opensky/kwil-db-sync-server

When the DB Sync extension is initialised it communicates with the Sync Server to restore any actions that exist on Arweave for that DB ID. It does this using the Bundlr GraphQL API and the server executes any found actions on the local DB. This enables actions to be re-played when a server has been taken down or when the schema is re-deployed on another server.

Once the initial actions have been restored, the Sync Server will continue to listen for transactions on the GQL API. This enables another Provider to run on a different server and submit actions to the same DB. By running 2 or more Providers with this service, specific databases can remain synced. If all Providers were to be destroyed, there is still the option to fully restore the DB, given the permanent storage of actions on Arweave.

Each action that is synced must have a private equivalent so that the data can be restored by the DB Owner / Admin only. This is as certain information such as @caller, $date, $id etc could not be replayed successfully at another time by different account.

Once the save_action method is called on the extension the data is saved to Arweave and can be found through the Bundlr GQL API.

The file stored on Arweave contains information about which action to run and what parameters to use:

Another server can deploy the same schema and remain in sync with the original DB by supplying the ID when registering the extension: