Giter Club home page Giter Club logo

aviondb's Introduction

AvionDB JS Implementation

AvionDB: A Distributed, MongoDB-like Database

Architecture

AvionDB Architecture

AvionDB uses OrbitDB stores to model MongoDB-like Databases. It creates a notion of:

See more details in the Specs.

Project status & support

Status: in active development

NOTE! AvionDB is alpha-stage software. It means AvionDB hasn't been security audited and programming APIs and data formats can still change. We encourage you to reach out to the maintainers if you plan to use AvionDB in mission critical systems.

This is the TypeScript implementation and it works with Browsers, Node.js & TypeScript with support for Linux, OS X, and Windows . The minimum required version of Node.js is now 8.6.0 due to the usage of ... spread syntax. LTS versions (even numbered versions 8, 10, etc) are preferred.

Table of Contents

Install

This module uses node.js, and can be installed through npm:

Using NodeJS

// Using npm
npm install --save aviondb

// Using Github
npm install git+https://github.com/dappkit/aviondb.git

We support both the Current and Active LTS versions of Node.js. Please see nodejs.org for what these currently are. The minimum required version of Node.js is now 8.6.0 due to the usage of ... spread syntax. LTS versions (even numbered versions 8, 10, etc) are preferred.

In a web browser

through Browserify

Same as in Node.js, you just have to browserify to bundle the code before serving it.

Note: The code uses es6, so you have to use babel to convert the code into es5 before using browserify.

through webpack

Same as in Node.js, you just have to webpack to bundle the the code before serving it.

Note: The code uses es6, so you have to use babel to convert the code into es5 before using webpack.

from CDN

Instead of a local installation (and browserification) you may request a remote copy of AvionDB from unpkg CDN.

To always request the latest version, use the following:

<!-- loading the minified version -->
<script src="https://unpkg.com/aviondb/dist/aviondb.min.js"></script>

CDN-based AvionDB provides the AvionDB constructor as a method of the global window object. Example:

// create an AvionDB instance
const aviondb = await AvionDB.init("DatabaseName", ipfs);

See examples/index.html to check out a fully functioning example for AvionDB with browser.

Usage

We are upgrading AvionDB to newer versions of js-ipfs. For now, please use [email protected]

Example

// Import modules
import AvionDB from "aviondb";
import IPFS from "ipfs";

const runExample = async () => {
  const ipfs = await IPFS.create();

  // Creates a db named "DatabaseName" in the ".aviondb" directory in the project root.
  // If no path option is defined, $HOME/.aviondb is used for the database directory (e.g. "C:/Users/John/.aviondb" or "~/.aviondb").
  const aviondb = await AvionDB.init("DatabaseName", ipfs, {
    path: "./.aviondb",
  });

  // Returns the List of database names
  await AvionDB.listDatabases();
  // prints ['DatabaseName']

  // Creates a Collection named "employees"
  const collection = await aviondb.initCollection("employees");

  // Returns the List of collection names
  await aviondb.listCollections();
  // prints ['employees']

  // Adding an employee document
  await collection.insertOne({
    hourly_pay: "$15",
    name: "Elon",
    ssn: "562-48-5384",
    weekly_hours: 100,
  });

  // We also support multi-insert using collection.insert()
  // See https://github.com/dappkit/aviondb/blob/master/API.md

  // Search by a single field Or many!
  var employee = await collection.findOne({
    ssn: "562-48-5384",
  });

  // We also support find(), findById()
  // See https://github.com/dappkit/aviondb/blob/master/API.md

  // Returns the matching document
  console.log(employee);
  // Prints the above added JSON document

  // Update a document
  var updatedEmployee = await collection.update(
    { ssn: "562-48-5384" },
    { $set: { hourly_pay: "$100" } }
  );

  // We also support updateMany(), findOneAndUpdate()
  // See https://github.com/dappkit/aviondb/blob/master/API.md

  // Returns the updated document
  console.log(updatedEmployee);
  // Prints the updated JSON document

  // await collection.close(); // Collection will be closed.
  // await aviondb.drop(); // Drops the database
  // await aviondb.close(); // Closes all collections and binding database.
  // await ipfs.stop();
};

runExample();

See FAQs Section for more examples on how to use AvionDB in your project.

API

See API.md for the full documentation.

Development

Run Tests

npm test

Benchmarks

Run Write Benchmark

npm run benchmarks:write

Run Query Benchmark

npm run benchmarks:query

Run Update Benchmark

npm run benchmarks:update

See benchmarks/ for more info on benchmarks.

Specs

We are working on the initial Specs. See AvionDB Specs doc

Projects using AvionDB

  • nOS Client: nOS is a blockchain powered virtual operating system that serves as the gateway to Decentralized Applications. The nOS Client (Developer MVP Release) allows for the development of Decentralized Applications that interact with Smart Contracts in the back-end.

  • orbitdb-nos-identity-provider: OrbitDB and AvionDB Identity Provider for nOS Network and other ARK Core-based blockchains.

  • bitsong-media-player: [Private Repo] BitSong Media Player for BitSong Blockchain, uses AvionDB to store, and share song tracks & metadata among the browser clients.

  • js-pinza: Pinza is a IPFS pinning system using orbit-db, and aviondb as a distributed database.

  • Movie-Reviews-CRUD-Application: A Kickstarter Application to understand how AvionDB Works by Implementing one.

Frequently Asked Questions

Are there implementations in other languages?

We are working to implement AvionDB for following languages:

  • NodeJS & Browser JS
  • Typescript
  • Golang
  • Rust

The best place to find out what is out there and what is being actively worked on is likely by asking in the Discord.

If you want or are planning to create an implementation in a language that is not listed here, then feel free to reach us out and discuss about it in the Discord.

Where can I see your Roadmap?

You can find our Roadmap here. The features in the Roadmap are taken from 2 separate issues(#7, #8) which individually maintain a list of feature proposals related to OrbitDB-specific improvements & AvionDB-specific improvements respectively.

The Roadmap is an open discussion, feel free to add your suggestions, comments.

What mongodb features does aviondb support?

You can find all the supported MongoDB-like features in our API docs.

How can I use AvionDB in my Application?

You can see the following examples to get started with AvionDB:

Other Questions?

If you didn't find the answer to your question(s), feel free to reach us out on Discord.

Contributing

Take a look at our organization-wide Contributing Guide.

As far as code goes, we would be happy to accept PRs! If you want to work on something, it'd be good to talk beforehand to make sure nobody else is working on it. You can reach us on Discord, or in the issues section.

If you want to code but don't know where to start, check out the issues labelled "help wanted", "discussion".

Please note that we have a Code of Conduct, and that all activity in the @dappkit organization falls under it. Read it when you get the chance, as being part of this community means that you agree to abide by it. Thanks.

Sponsors

The development of AvionDB has been sponsored by:

If you want to sponsor developers to work on AvionDB, please consider sponsoring using the "Sponsor" button on the top of the AvionDB Github Page.

License

MIT

aviondb's People

Contributors

deanpress avatar ethdev279 avatar vasa-develop avatar vaultec81 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aviondb's Issues

Feature list for AvionDB

This issue represents a list of features that will be worked on in the near future releases.

Note that this is just a list of features. Any feature that is decided to be worked upon, will have its own separate issue.

Feature Format


  • Feature Title ( Link to related issue/PR )
Add some context to clarify the feature in question.

  • Here if the checkbox is checked, then that means this feature is/has been worked upon.
  • Each checked feature should have a link to the related issue/PR where it is being discussed.
  • Each feature should have a description that clarifies the context of the feature in question.

Feature List

  • File Management Support (#5)
    Out-of-the-box support for adding/fetching/removing files using AvionDB.

  • Access Control
    Access control for managing authentication, authorization between AvionDB (OrbitDB) peers.

  • Secure Pub-Sub
    Out-of-the-box encryption over pub-sub for secure communication between AvionDB (OrbitDB) peers.

  • Store (Collection) Sharding
    In some cases, when we store a lot of data within the same Collection, the performance (read/write/replicate) of AvionDB can become worse. A possible method to mitigate this could be sharing the Collections and managing the set of "shard" Collections to give a notion of a single Collection.

  • Add support for events
    Events on insertion, deletion, and set of a document. Non-essential feature, a "nice to have" feature.

  • GUI Dashboard to visualize & interact with the data in AvionDB.
    A GUI dashboard like Mongo Compass for AvionDB. Existing Resources

  • An optional, out-of-the-box pinning service for persistent data storage on AvionDB.
    There are 2 options here:

  • OrbitDB community is starting a public Orbitdb store pinning service: nebulousity, for general users.

  • Dappkit will have its own Orbitdb store pinning service for production use-cases.

  • Add support for common Cursor methods (#12)
    Change the API signatures to allow passing cursor method options & add support for limit.

  • Add support for "Transactions"
    Add Transactions support for atomicity, sessions.

  • Add support to list databases created by AvionDB (#801711)
    Add a static function like AvionDB.listDatabases() to list the Databases.

  • Support for Schema for AvionDB (#23)
    Support for MongoDB-like Schemas.

Typescript Interfaces & ESLint Config

It'd be great to see the AvionDB environment fully integrated with Typescript (interfaces for each class etc, strict types for method arguments etc).

I also noticed a lot of variables use var instead of const or let. An ESLint config could help enforce some requirements (such as only use const or let instead of var , only allow arrow functions etc)

Release v0.1.0 ๐Ÿš€

What's left for release

  • Update documentation to include latest API specs.
  • Create a change log. Although won't be used until later releases.
  • Add MIT license to readme and repo.
  • Different non-placeholder name for the project.
  • Add an Overview of the project Architecture & description in README
  • Add Usage section in README

Additional tasks:

  • Contributors & thank yous list
  • Contributing Guidelines
  • Code of Conduct
  • Create pre-release & test

Estimated shipping date

Saturday the 18th of April

General bugfixes and updates:

Relevant PRs

  • feat: add support for query operators (#2)
  • refactor: optimise findOne method (#1)
  • Create LICENSE (#4)

This release is not finalized yet additional changes prior to release may be added. Considerable amount of review is required before finalization.

Reviewers and contributions are welcome.

project status?

Is this abandoned? Assimilated into OrbitDB? Blocked by something?

AvionDb Roadmap April-May-Beyond

Roadmap

Within Next two months

  • Exposing pubsub channel to orbit-db-store interface
    • Secure, authenticated pubsub (Aviondb specific, or orbitdb specific?)
    • Non Identity based authentication. Tokens, symmetric keys, 3rd party oauth etc. Goes along with custom access controller.
  • Plugins. Non essential additions for authentication, data manipulation, secondary-higher level Peer-To-Peer communications.
  • Custom access controller. (Track Progress here)
  • Server daemon
  • CLI, import/export other operations if possible

Two months and beyond

  • Store (Collection) Sharding
  • File Management Support (#5). Start of tmfs standard.
  • Full scale mongodb TCP, HTTP API for 3rd party applications.

See supporting issues
AvionDb feature list
Orbit-db feature requirements

Why NodeJS & Browser JS, Typescript, Golang and Python over Rust?

It seems to me like a plan that implements AvionDb in NodeJS & Browser JS, Typescript, Golang and Python requires writing code 3-4 times and at the end of the day the AvionDb still won't be able to be easily used on Native Android/Native iOS/Xamarin/Flutter. If the code was written in Rust it could be used in all-contexts (maybe a bit additional work on the web to integrate with IndexedDB)

At the same time both performance and security are likely worse then if the project would be in Rust. Both are important for this project.

Feature Requirements of Orbit-db

Suggestions:

What needs to be added, related issues.

  • Exposing pubsub channel to orbit-db-store interface (https://github.com/orbitdb/orbit-db/issues/784)
  • Orbitdb instance pass through to orbit-db-store interface.
  • Handle the process of connecting to IPFS swarm peer & starting pub-sub channel inside some custom method. Bootstrap peers through IPFS DHT or other mechanism in a decentralized manner.

Fulfillments:

Keep track of PRs.

[BUG]: Cannot list or get existing collection instance after dapp restart/refresh

Version:

0.1.1

Platform:

NodeJS & Browser

Describe the bug
The way we track the open collections is not persistent.

To Reproduce
Steps to reproduce the behavior:

  1. Create a collection named "users" in a browser.
  2. Refresh the browser.
  3. Try listing the collections using db.listCollections().
  4. Try getting the instance of the previously created collection using db.collection("users")

Expected behavior

  • Step 3 should list the previously created collections: ["users"]
  • Step 4 should return a "users" collection instance.

Observed Behaviour

  • Step 3 gives an empty array: [] meaning no open collections.
  • Step 4 throws error: Collection: users is not open.

Support for Schema for AvionDB

A Schema is a JSON object that allows you to define the shape and content of documents and embedded documents in a collection. You can use a schema to require a specific set of fields, configure the content of a field, or to validate changes to a document based on its beginning and ending states.

This issue is for discussing various approaches to implement Schema for AvionDB. The goal here is to pool different views on:

  • Why do we need Schema?
  • How we can implement Schema for AvionDB? (at least a rough idea)

This issue would be resolved to create a new issue with technical specifications for implementing Schema model.

This is an open discussion, feel free to add your comments, suggestions.

AvionDB.init open remote database

Is your feature request related to a problem? Please describe.
As stated in the docs, AvionDB.init is preferred over open or create. Thus I expect that init would open an existing database if given the address and create or open database if given a name. However when given the address, say zdpuAzMS57cFLjf2iiekP964FtCepYA1VzmZ5rHkrqsGfDs98/test, the init function throws an error saying Given database name is an address. Please give only the name of the database!

Describe the solution you'd like
The init function should open a remote database if given the address.

Describe alternatives you've considered
If open should not be replaced by init in common usage, the API docs should delete the line NOTE: It is recommended to use init instead of create & open for better developer experience.

Additional context
None

GraphQL API for AvionDB

Why do you even need one?

Now, the thing is that when a request is made to AvionDB, it returns ALL the data in that document, whether needed or not. This is especially annoying in apps where you have Images and stuff which is all base64; here, all the base64 data is returned, even if it's not needed at all.
And that's just one situation, there are many many more similar ones...

Example: MongoDB's API for GraphQL

Further Typescript support

Now that the next release is ported to Typescript, existing classes and methods should be properly typed and documented with jsdoc so that Intellisense can guide development. That should make working with AvionDB in VSCode a breeze!

Support for managing files with AvionDB

The agenda of this issue is to come up with a rough specification of the store explaining what methods/features we need in the custom store & how will the store work.

The basic idea here is to create a custom store for managing big files in IpfsDB.

A few things that are desirable in such a custom store:

  • It should support methods to add, read, list, remove the files.
  • Should store metadata for each file.
  • Add more...

How will this work?

We can add the files to the connected IPFS instance, get the hash and other metadata and store the details in the custom store.

This is an open discussion, feel free to comment your suggestions.

Lack of video tutorials for the implementation of AvionDB

Is your feature request related to a problem? Please describe.
Lack of video tutorials for the implementation of AvionDB, this drives away potentially 100s of people who find the software useful, but (due to the lack of knowledge and their poor understanding of the docs) don't end up using it; but rather end up using centralised services like firebase.

Describe the solution you'd like
I'd Like more video tutorials, on how to implement AvionDB In A React-Native Application and How Can AvionDB replace firebase and all.

Describe alternatives you've considered
I've considered the docs; but, I being a new developer, didn't really understand the docs fully.

Additional context
It'd Be great if the tutorials also include a little design, that makes it far more fun to watch.

[BUG]: version is undefined

Version:
0.2.3

Platform:
Browser (firefox)

Describe the bug
I cannot seem to get a minimal example of AvionDB to work. I am using VueJS, but I don't think the error is related to that.

To Reproduce
Here is my VueJS script in the main component:

<script>
const IPFS = require('ipfs')
import AvionDB from 'aviondb'

export default {
  name: 'HelloWorld',
  props: {
    msg: String
  },
  mounted: function() {
    const runExample = async () => {
    const ipfs = await IPFS.create();

    // Creates a db named "DatabaseName" in the ".aviondb" directory in the project root.
    // If no path option is defined, $HOME/.aviondb is used for the database directory (e.g. "C:/Users/John/.aviondb" or "~/.aviondb").
    const aviondb = await AvionDB.init("DatabaseName", ipfs, {
      path: "./.aviondb",
    });

    // Returns the List of database names
    await AvionDB.listDatabases();
    // prints ['DatabaseName']

    // Creates a Collection named "employees"
    const collection = await aviondb.initCollection("employees");

    // Returns the List of collection names
    await aviondb.listCollections();
    // prints ['employees']

    // Adding an employee document
    await collection.insertOne({
      hourly_pay: "$15",
      name: "Elon",
      ssn: "562-48-5384",
      weekly_hours: 100,
    });

    // We also support multi-insert using collection.insert()
    // See https://github.com/dappkit/aviondb/blob/master/API.md

    // Search by a single field Or many!
    var employee = await collection.findOne({
      ssn: "562-48-5384",
    });

    // We also support find(), findById()
    // See https://github.com/dappkit/aviondb/blob/master/API.md

    // Returns the matching document
    console.log(employee);
    // Prints the above added JSON document

    // Update a document
    var updatedEmployee = await collection.update(
      { ssn: "562-48-5384" },
      { $set: { hourly_pay: "$100" } }
    );

    // We also support updateMany(), findOneAndUpdate()
    // See https://github.com/dappkit/aviondb/blob/master/API.md

    // Returns the updated document
    console.log(updatedEmployee);
    // Prints the updated JSON document

    // await collection.close(); // Collection will be closed.
    // await aviondb.drop(); // Drops the database
    // await aviondb.close(); // Closes all collections and binding database.
    // await ipfs.stop();
  };

  runExample();

  }
}
</script>

Expected behavior
I would expect the DB to initialise and log to console as instructed.

Observed behavior
I get the following error message in the console:

Uncaught TypeError: version is undefined
    <anonymous> use-native.js:4
    <anonymous> use-native.js:12
    js chunk-vendors.js:1578
    __webpack_require__ app.js:849
    fn app.js:151
    <anonymous> index.js:6
    js chunk-vendors.js:1512
    __webpack_require__ app.js:849
    fn app.js:151
    <anonymous> index.js:5
    js chunk-vendors.js:1590
    __webpack_require__ app.js:849
    fn app.js:151
    <anonymous> EnvironmentAdapter.js:5
    js chunk-vendors.js:422
    __webpack_require__ app.js:849
    fn app.js:151
    <anonymous> index.js:19
    <anonymous> index.js:92
    js chunk-vendors.js:457
    __webpack_require__ app.js:849
    fn app.js:151
    <anonymous> index.js:3
    js chunk-vendors.js:493
    __webpack_require__ app.js:849
    fn app.js:151
    <anonymous> cjs.js:5
    ./node_modules/cache-loader/dist/cjs.js?!./node_modules/babel-loader/lib/index.js!./node_modules/cache-loader/dist/cjs.js?!./node_modules/vue-loader/lib/index.js?!./src/components/HelloWorld.vue?vue&type=script&lang=js& app.js:950
    __webpack_require__ app.js:849
    fn app.js:151
    <anonymous> HelloWorld.vue:1
    ./src/components/HelloWorld.vue?vue&type=script&lang=js& app.js:1112
    __webpack_require__ app.js:849
    fn app.js:151
    <anonymous> HelloWorld.vue:1
    vue app.js:1100
    __webpack_require__ app.js:849
    fn app.js:151
    <anonymous> cjs.js:2
    ./node_modules/cache-loader/dist/cjs.js?!./node_modules/babel-loader/lib/index.js!./node_modules/cache-loader/dist/cjs.js?!./node_modules/vue-loader/lib/index.js?!./src/App.vue?vue&type=script&lang=js& app.js:938
    __webpack_require__ app.js:849
    fn app.js:151
    <anonymous> App.vue:1
    ./src/App.vue?vue&type=script&lang=js& app.js:1053
    __webpack_require__ app.js:849
    fn app.js:151
    <anonymous> App.vue:1
    vue app.js:1041
    __webpack_require__ app.js:849
    fn app.js:151
    <anonymous> main.js:11
    js app.js:1148
    __webpack_require__ app.js:849
    fn app.js:151
    1 app.js:1161
    __webpack_require__ app.js:849
    checkDeferredModules app.js:46
    <anonymous> app.js:925
    <anonymous>

Example of AvionDB with chat app?

Note: I hope this is the correct place to post this (if not, please let me know where!)

I was wondering what the correct IPFS/AvionDB implementation of a basic chat app would be. In server-based MongoDB, I would have:

  1. A database with usernames, passwords (hashed, of course), and roles
  2. A database of messages, with content and username

Then:

  • On registration, the appropriate row would be added to the user database, which would be used for authentication
  • Every registered user would have permission to add, delete or modify messages in the message database (with their username), but only users with admin role would have permission to edit or delete messages from any username.

Is it at all possible to do something similar with a p2p database such as AvionDB? Or is there any fundamental philosophical difference that I should adapt to?

Many thanks!

Empty query operators does not return a result.

Version:
0.1.1
Platform:
NodeJS & Browser

Describe the bug
Methods using query operators should return all results. In order to get any results at all a query operator must be supplied.

To Reproduce
Steps to reproduce the behavior:

  1. Run Collection.find({}), collection.findOne({}) with an empty object as query.
  2. Result is [] (an empty array) or {} (an empty object).

Expected behavior
Expected behavior for Collection.find({}) should be a full array containing results, and collection.findOne({}) should return the first object found.

Observed behavior
Result is [] (an empty array) or {} (an empty object).

Add support for cursor methods

Is your feature request related to a problem? Please describe.
Allowing cursor methods like limit

Describe the solution you'd like
Changing the API signatures to support passing cursor options. For eg, changing to find method to look like this find(query, projection, options, callback)

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

Golang implementation of AvionDB

A couple of weeks ago I asked the Berty guys (@glouvigny) about feature parity of their OrbitDB's Golang implementation & official JS implementation.

(Our conversation from Discord)

Hi! I know that go-ipfs-log is compatible with the JS version as we are using the same test suite (same binary content, same CIDs).

However regarding go-orbit-db cross language compatibility has not been our focus nor tested so far, we have an open issue here berty/go-orbit-db#9

I don't know your needs in terms of access controllers and data stores, but we're far from feature parity on those fields. You will only find an IPFS access controller and the EventLog, KeyValue stores

@vaultec81 We can explore https://github.com/berty/go-orbit-db whenever we find some time to see how we can go upon implementing AvionDB in Golang.

This is an open discussion. Feel free to add your comments/suggestions.

Will AvionDb support multi-document atomic transactions?

Many databases have atomic transactions. That's useful because it prevents you from bringing your data into a bad state when a part of a transaction that goes over multiple document fails.

Will AvionDb support multi-document atomic transactions or will it in that regard be like MongoDb which doesn't?

Discussion: Implementing B-Tree to improve query performance

How did we add the b-tree implementation?

The current implementation of b-tree is really minimal. We did 2 changes:

  1. Populate the b-tree when we write anything to the store. This is done in the handleInsert method in src/CollectionIndex.js.
    NOTE that we still populate the data in the ipfs-log, as we plan to use it as persistent storage from which we can load the data to the b-tree whenever we import an existing database.
    Below, in the "Observations" section we talk about if populating b-tree in the handleInsert method has any overhead on the write operations or not.

  2. We replaced the logic of findById in src/CollectionIndex.js. Instead of looping through all the Log entries, we now just query the b-tree.

Benchmark Results

There are 2 branches master, feat/b-tree. The master branch works without the b-tree and feat/b-tree works with b-tree.

You can check out the query benchmark you can use the following script:

npm run benchmark:query

Benchmark Results

In master branch

Creating 5000 documents for querying. Stand by! 
1977 queries per second, 1977 queries in 1 seconds (Oplog: 5000)
1938 queries per second, 3915 queries in 2 seconds (Oplog: 5000)
1909 queries per second, 5824 queries in 3 seconds (Oplog: 5000)
2024 queries per second, 7848 queries in 4 seconds (Oplog: 5000)
2023 queries per second, 9871 queries in 5 seconds (Oplog: 5000)
1963 queries per second, 11834 queries in 6 seconds (Oplog: 5000)
1951 queries per second, 13785 queries in 7 seconds (Oplog: 5000)
1969 queries per second, 15754 queries in 8 seconds (Oplog: 5000)
1927 queries per second, 17681 queries in 9 seconds (Oplog: 5000)
--> Average of 1970.3 q/s in the last 10 seconds

In feat/b-tree branch

Creating 5000 documents for querying. Stand by! 
813699 queries per second, 813699 queries in 1 seconds (Oplog: 5000)
816660 queries per second, 1630359 queries in 2 seconds (Oplog: 5000)
783972 queries per second, 2414331 queries in 3 seconds (Oplog: 5000)
774793 queries per second, 3189124 queries in 4 seconds (Oplog: 5000)
784069 queries per second, 3973193 queries in 5 seconds (Oplog: 5000)
801397 queries per second, 4774590 queries in 6 seconds (Oplog: 5000)
839060 queries per second, 5613650 queries in 7 seconds (Oplog: 5000)
834974 queries per second, 6448624 queries in 8 seconds (Oplog: 5000)

Observations

As we mentioned above, I ran our "write" benchmarks to check for any significant overheads.

You can run "write" benchmarks using the following script:

npm run benchmark:write

In master branch

682 writes per second, 682 writes in 1 seconds (Oplog: 682)
822 writes per second, 1504 writes in 2 seconds (Oplog: 1504)
846 writes per second, 2350 writes in 3 seconds (Oplog: 2350)
895 writes per second, 3245 writes in 4 seconds (Oplog: 3246)
956 writes per second, 4201 writes in 5 seconds (Oplog: 4201)
853 writes per second, 5054 writes in 6 seconds (Oplog: 5054)
764 writes per second, 5818 writes in 7 seconds (Oplog: 5818)
708 writes per second, 6526 writes in 8 seconds (Oplog: 6526)
562 writes per second, 7088 writes in 9 seconds (Oplog: 7088)
--> Average of 787.6 q/s in the last 10 second

In feat/b-tree branch:

649 writes per second, 649 writes in 1 seconds (Oplog: 649)
764 writes per second, 1413 writes in 2 seconds (Oplog: 1413)
836 writes per second, 2249 writes in 3 seconds (Oplog: 2249)
856 writes per second, 3105 writes in 4 seconds (Oplog: 3105)
915 writes per second, 4020 writes in 5 seconds (Oplog: 4020)
933 writes per second, 4953 writes in 6 seconds (Oplog: 4954)
966 writes per second, 5919 writes in 7 seconds (Oplog: 5919)
883 writes per second, 6802 writes in 8 seconds (Oplog: 6802)
903 writes per second, 7705 writes in 9 seconds (Oplog: 7705)
--> Average of 861.9 q/s in the last 10 seconds

As we can see, the performance of write operation is still comparable (the write benchmark of feat/b-tree even exceeded than that of master, due to some intermittent behavior). So, even if we populate b-tree in addition to populating the Log entries, there is not much harm to the write performance.

Open Questions

  1. How to deal with replicate events?
  2. How to marry b-tree with orbit-db-store?
  3. This is a moonshot, but the time complexity of b-tree operations is the following:
Search		O(log n)	O(log n)
Insert		O(log n)	O(log n)
Delete		O(log n)	O(log n)

How does this compare to the current write time complexity in ipfs-log? We did recently got a greater write performance with @phillmac's batch put implementation...but can a b-tree based ipfs-log give a better write performance too?

This is an open discussion, feel free to add more questions & suggestions. GO CRAZY!

Transpiling for the Typescript release (0.2.3)

Before the Typescript version can be released, there needs to be proper transpiling of the package.

Ideally, Webpack + Babel OR Rollup + Rollup-Typescript is used to transpile the project into a minified JS package that can be imported both in the browser and in Node.

Release v0.2.4 ๐Ÿš€

What's left for the release

  • Saving files on AvionDB
  • Browser Tests

Features for future releases

  • Hosted Backup using js-pinza
  • Support for Schemas
  • Server Daemon

Estimated shipping date

TBD

General bugfixes and updates

Relevant PRs

  • Todo

This release is not finalized yet additional changes prior to release may be added. A considerable amount of review is required before finalization.

Reviewers and contributions are welcome.

[BUG]: OpenError: IO error: lock orbitdb/Qm.../keystore/LOCK: already held by process

Version:
v0.2.0

Platform:
node.js

Describe the bug
On creating a database using [email protected], it throws the following error:

OpenError: IO error: lock orbitdb/Qm.../keystore/LOCK: already held by process

To Reproduce
Steps to reproduce the behavior:

  1. Install [email protected], [email protected]
  2. Run the usage example

Expected behavior
It should create a database.

Observed behavior
We get an error.

Screenshots
error-avion

Desktop (If applicable, please complete the following information):

  • OS: Ubuntu 16.04

[BUG]: "Database already exists" and websockets errors when reloading the page with your sample code

Version:
Current from CDN

Platform:
Browser

Describe the bug
I am getting the following errors when I reload the browser page running your sample browser code from: https://simpleaswater.com/intro-to-aviondb/

Numerous instances of:

WebSocket connection to 'ws://127.0.0.1:8081/p2p/QmSoLPppuBtQSGwKDZT2M73ULpjvfd3aZ6ha4oFGL1KrGM' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED
t.exports @ index.min.js:2
_connect @ index.min.js:2
dial @ index.min.js:2
dial @ index.min.js:2
dialAction @ index.min.js:2
(anonymous) @ index.min.js:2

WebSocket connection to 'ws://127.0.0.1:8081/p2p/QmSoLueR4xBeUbY9WZ9xGUUxunbKWcrNFTDAadQJmocnWm' failed: WebSocket is closed before the connection is established.
(anonymous) @ index.min.js:2
c.close @ index.min.js:2
o @ index.min.js:2
r @ index.min.js:2
(anonymous) @ index.min.js:2
run @ index.min.js:2
async function (async)
run @ index.min.js:2
_createPendingDial @ index.min.js:2
connectToPeer @ index.min.js:2
_maybeConnect @ index.min.js:2
_onDidStart @ index.min.js:2
start @ index.min.js:2
async function (async)
start @ index.min.js:2
(anonymous) @ index.min.js:2
async function (async)
(anonymous) @ index.min.js:2
(anonymous) @ index.min.js:2
create @ index.min.js:2
async function (async)
create @ index.min.js:2
runExample @ aviondb.html:10
(anonymous) @ aviondb.html:67

Followed by:

Uncaught (in promise) Error: Database '/orbitdb/zdpuAn2qsM6P9E3DKz1hL4pUEaJwGqUh7LEX4eCnzgwABoeZV/employees' already exists!
    at k.create (aviondb.min.js:25)
create @ aviondb.min.js:25
async function (async)
runExample @ aviondb.html:10
(anonymous) @ aviondb.html:67

It would seem to me that by using aviondb.initCollection("employees"); it should simply open the existing collection, not try to create a new one.

Also, it would seem to me that by using await aviondb.drop(); the collection would be removed along with the database.

What else is notable here, is if I delete the databases in IndexedDB upon page reload, the issue does not happen. But that causes me to wonder if it's because my ipfs node ID is changing in that scenario and therefore I no longer can access the previously created database and collection.

TypeError: Store is not a constructor

Version:
0.2.3

Platform:
NodeJS

Describe the bug

TypeError: Store is not a constructor
      at OrbitDB._createStore (.\node_modules\orbit-db\src\OrbitDB.js:231:19)
      ...snip, implementation specific traceback...

To Reproduce
Steps to reproduce the behavior:

  1. Install aviondb from npm
  2. require("aviondb")
  3. open a collection directly using orbitdb.open()

Expected behavior
No error should be thrown. Aviondb instance should open.

Observed behavior
Error thrown when opening an aviondb instance

Additional context
In my use case a collection was opened directly instead of going through aviondb's binding database. This worked prior to 0.2.3, however it no longer works after upgrade. Possible regression.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.