This issue serves as preparation to implement:
https://www.notion.so/worldbrain/StorexHub-Interface-3a6024e9538e404ab4df41771302de3f

A rough first draft of the tasks involved (to be edited and estimated as the discussion evolves)

Base:

• Implement standard plugin directory living next to the binary (4 hours)
• Implement endpoint to get information about available and installed plugins (3 hours)
• Enabled/disable plugins? (3 hours)

UI:

• Build & serve the front-end from a certain URL (4 hours, requires Webpack setup, etc.)
• Start the front-end on first startup (3 hours)
• Plugin overview UI (2 days)
• Plugin settings UI (4 days)
  • Standard settings
  • Links to custom settings
• App functions (index page in Memex, get tag suggestions, etc.) (2 days)
• System tray (3+ days, very unpredictable, no good solutions)

Plugins & updates:

• Safe stopping of plugin, so new version of it can be loaded (2+ days)
• GitHub-based plugin discovery (for now only featured plugins) (4+ days)
• Checking for plugin updates (4+ days)
• Plugin reload mechanism, building on safe stopping of plugin (1 day)
• Plugin update mechanism (means downloading, overwriting, reloading.) (2 days)
• Plugin change detection and communicating this to the UI (2+ days, depends suitability of file watchers, easy of distribution of native Node.js modules.,)
• Ideal: zip-file support for plugins (2 days)

I tried to install Storex-Hub on my Mac per the instructions here and the terminal started screaming at me when I tried npm install lol

Is Storex-Hub currently functional? I'm trying to get Memex to sync up with DevonThink and Storex seemed like the most natural route.

npm ERR! code ERESOLVE
npm ERR! ERESOLVE could not resolve
npm ERR!
npm ERR! While resolving: @worldbrain/[email protected]
npm ERR! Found: @worldbrain/[email protected]
npm ERR! node_modules/@worldbrain/storex
npm ERR!   @worldbrain/storex@"^0.4.1" from the root project
npm ERR!   @worldbrain/storex@"^0.4.1" from [email protected]
npm ERR!   node_modules/storex-hub-plugin-internal-selftest
npm ERR!     dev storex-hub-plugin-internal-selftest@"^0.0.2" from the root project
npm ERR!   3 more (@worldbrain/storex-backend-typeorm, ...)
npm ERR!
npm ERR! Could not resolve dependency:
npm ERR! peer @worldbrain/storex@"^0.3.5" from @worldbrain/[email protected]
npm ERR! node_modules/@worldbrain/storex-backend-dexie
npm ERR!   @worldbrain/storex-backend-dexie@"^0.3.1" from the root project
npm ERR!   peer @worldbrain/storex-backend-dexie@"^0.3.1" from @worldbrain/[email protected]
npm ERR!   node_modules/@worldbrain/storex-middleware-change-watcher
npm ERR!     @worldbrain/storex-middleware-change-watcher@"^0.1.1" from the root project
npm ERR!
npm ERR! Conflicting peer dependency: @worldbrain/[email protected]
npm ERR! node_modules/@worldbrain/storex
npm ERR!   peer @worldbrain/storex@"^0.3.5" from @worldbrain/[email protected]
npm ERR!   node_modules/@worldbrain/storex-backend-dexie
npm ERR!     @worldbrain/storex-backend-dexie@"^0.3.1" from the root project
npm ERR!     peer @worldbrain/storex-backend-dexie@"^0.3.1" from @worldbrain/[email protected]
npm ERR!     node_modules/@worldbrain/storex-middleware-change-watcher
npm ERR!       @worldbrain/storex-middleware-change-watcher@"^0.1.1" from the root project
npm ERR!
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force, or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
npm ERR!
npm ERR! See /Users/roberthaisfield/.npm/eresolve-report.txt for a full report.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/roberthaisfield/.npm/_logs/2022-04-28T18_28_15_175Z-debug-0.log

I'm assuming that storex-hub would have a similar license to [storex] (i.e. MIT) but having that ratified and:

• added to package.json
• present in a LICENSE file
• flagged in repository settings

would be killer.

To make early adoption by developers easier, it should be possible use Storex Hub with existing plugins as easily as possible. A first version of the experience could be:

• User clones master branch of storex-hub or storex-hub-gui-nwjs
• User runs command to install plugins, using NPM under the hood

The base work for this is already done, but needs some improvements. Namely, right now different can register collections by naming their collections <app name>:<collection name>. This 1) makes it more difficult for users to adapt their existing code written for a single-user experience because all of their collection name references need to change, and 2) even though it's short, we want to prevent name collisions by adopting a reverse-domain naming scheme.

Tasks:

• Change executeOperation call to automatically start queries in an apps own tables
• Improve schema building function, which currently just meshes the schemas of all different applications together.

Almost every app needs to store media, such as images, documents, etc. Storex Hub should be able to store and manage this way, addressing some common issues such as tracking file ownership, access rights and allowing the user to manage and use their media library.

I'm assuming that storex-hub would have a similar license to [storex] (i.e. MIT) but having that ratified and:

• added to package.json
• present in a LICENSE file
• flagged in repository settings

would be killer.

Hi!

I really love the idea of the project! 😊

Currently, I am using Singer Taps & Targets + Meltano to achieve similar functionality.

There also seem to be build some other projects in similar vain building on the Singer specification.

How do you see the relationship of this project to Singer? 🙂

Cheers!

As apps evolve, they will change their schema. They could migrate their data themselves, but the problem is that other apps might rely on older versions of their schema. Therefore, using the storex-schema-migration package, we should allow users to describe their migrations in a declarative way. This means that 1) Storex Hub can execute them however it wants, and 2) it may be possible to automatically translate queries on older schemas to queries for the new schema (to be investigated). Another option would be delegate the responsibility for this backward compatibility to the apps themselves, but this would be sub-optimal for various reasons.