Giter Club home page Giter Club logo

airport's Introduction

AIRport - Decentralized Relational Platform

Description

AIRport allows Distributed Applications to interoperate by providing an API framework on top of a relational database. Data is stored in virual Repositories with one Transaction Log per repository.

More documentation can be found at Beyond Decentralized.

How it works

  • User navigates to a web page and saves/retrieves data.
  • A new tab is opened in the background with AIRport framework in it
  • AIRport contacts Transaction Log data hosts (Ex: IPFS), retrieves data, puts into in-memory SqLite database, queries it, makes modifications to it, writes the Transaction Log entries back to data hosts.
  • Access to database is allowed only from App logic that runs in domain-sanboxed IFrames (inside AIRport tab).

Build instructions

If something isn't building and it really should, run 'rush update'

Check out AIRroot for framework build instructions.

Entity Definitions

@Entity()
export class Parent extends AirEntity {

    value: string;

    total: number;

    @OneToMany({mappedBy: 'parent'})
    children: Child[];
}

@Entity()
export class Child extends AirEntity {

    value: string;

    @ManyToOne()
    parent: Parent;
}

Queries

Apps can define entities which depend on entities in other Apps, via @ManyToOne() relations. Apps can build joins that include entities from other Apps.

Data Access Objects (DAOs)

@Injected()
export class ParentDao 
       extends BaseParentDao {

    async findById(
      parentUuId: string
    ): Promise<Parent> {
      let p: QParent,
          c: QChild
      return await this._findOne({
        SELECT: {
          '*': Y,
      	  children: {}
        },
        FROM: [
          p = Q.Parent,
          c  = p.children.LEFT_JOIN()
        ],
        WHERE: p.equals(parentUuId)
      })
    }

    async updateInPlace(
        parent: Parent
    ): Promise<void> {
        const p: QParent
        await this._updateWhere({
            UPDATE: p = Q.Parent,
            SET: {
                total: PLUS(p.total, 1)
            },
            WHERE: p.equals(parent)
        })
    }

}

APIs

Public API methods are annotated with @Api() decorator. Apps can invoke @Api() methods of other Apps. Apps can join across application schemas but must call APIs of other Apps to modify their data.

@Injected()
export class ParentApi {

    @Inject()
    parentDao: ParentDao

    @Api()
    async save(
      parent: Parent
    ): Promise<void> {
      await this.parentDao.save(parent)
    }

    @Api()
    async findById(
      parentUuId: string
    ): Promise<Parent> {
      return await this.parentDao.findById(parentUuId)
    }

}

Directory Structure

ORMs Object Relational Mapping frameworks.

apis Internal and external APIs.

databases Database adapters.

engines Core Logic of AIRport.

generators Code generators.

libs Libraries.

platforms Platform adaptors for Web and Native.

schemas Internal AIRport schemas.

Latest

Find out about the latest developments at the Beyond Decentralized blog

License

AIRport is distributed under the terms of both the MIT license and the Apache License (Version 2.0).

See LICENSE-APACHE, LICENSE-MIT

airport's People

Contributors

artem-v-shamsutdinov avatar

Stargazers

 avatar  avatar

Watchers

 avatar

Forkers

cryptobuks

airport's Issues

Implement cross-subchain resolution

When users make concurrent changes online update conflicts make occur. This means that AIRport will have to:

  1. Compute where Repository's transaction log chains diverge and re-compute diffs from the root of the diversion. This will have to be done every time divergence is found to ensure consistent data results across all databases that have a given repository.
  2. Add conflict resolution based on sub-chains (on top of current one, based on reported timestamps of changes).

Move all SQL related functions and objects to @airport/tarmaq-sql

For ability to hot load:

Move all SQL related top level constants (Y, N, etc.) and functions (AND, OR, NOT, MAX, MIN, etc.) to a new library. Back their implementations with globaThis based injection references (allowing for version numbers):

export const Y = globalThis.IOC.getSQLObject('Y', '2.0.0')

export const AND = globalThis.IOC.getSQLFuction('AND', '2.0.0')

This should be done before hot loading is enabled (eventually). The @airport/tarmaq-sql will be included in every patch that has DAOs in it but will be just a pass though object to the underlying injectable functionality.

Implement Binary Object support

Need support for Binary Objects: Images, Videos, Files

Currently the idea is to:

  1. Save the objects (to storage)
  2. Get the address of the object and add it to the Repository
  3. Binary Objects are Promise types with a special property to get the address immediately
  4. Calling the promise returns the underlying data stream (that can be displayed via a data link).

Add support for in-framework-tab App UIs

Right now App UIs run in other tabs. Support for that prevents future sandboxing of UIs via QuickJS (need to investigate if this is really needed, maybe warnings to users are enough). For now support for running outside UIs should be available but apps should also be runnable inside Iframes (within AIRport framework tab).

Running Apps in their own tabs should still be supported.

This does not include custom cross-app navigation techniques (they would make it hard for UI developers to write their apps) so the UIs should just reload the content of the UI Iframe (but saves on RAM, which is constrained). This makes UI loading slower but there are already plenty of techniques to speed that up.

Graph Database - Disallow indexes on foreign keys

Once Graph Database compatibility is added (via optional on per link basis, internal Many-to-Many tables) this will be needed:

Many-to-many links foreign keys no longer reside in AIR tables themselves and are instead moved to the link tables. Thus there may not be any indexes on AIR table that include foreign key columns. With link implementation such indexes are omitted and DDL-creation-time but there should be a generation-time check for this.

Implement basic cross-repository navigation & autoloading

Current solution for cross repository navigation is partial only and is not automated:

  • When Repository records reference records in other repositories all records referenced via @manytoone links are copied into the current Repository.
  • Source Ids are kept in separate fields (to be able to later lookup original records)

For @ OneToMany links the current proposal is at:

https://beyond-decentralized.world/2022/Cross_Repository_OneToManys.html

Instead the new solution is to:

  1. Remove the source Id fields
  2. Create In-Repository ledge table for both @ ManyToOne and @ OneToMany links. Ledger table allows for efficient scanning across records.
  3. Save cross-repository references that point to other Repositories via @ MayToOne links. Save @ OneToMany references if they are across tables in the schemas because if there is a @ OneToMany then it may be used in queries.

References will be AirEntities themselves (thus pointing to the member Repository) and will contain the referenced Repository Id as well as the ApplicationRelation used to make that reference. There may be duplicates in these and no maintenance is performed on them - create only. This means that if a repository is no longer referenced then it will still be eagerly loaded - it shouldn't be a common occurrence but this might have to be changed in the future.

  1. At repository-loading time scan for referenced repositories and load them in the background.
  2. Wire in repository loading notifications to Observable queries and re-query on load (at repository loading time).

Load Repositories by processing query relations.

Process relations found in queries where Repository Ids are used as input. Using the CrossRepositoryRelationLedger deduce from the query relations the repositories that likely have to be loaded and load them (without loading their dependencies).

Investigate: Basic App and UI network restrictions

There is nothing wrong with apps using network APIs, as long as they don't abuse them (steal user data). In fact it could greatly enrich the ability of apps to provide functionality (because they be able to use additional data and data processing capabilities).

However there should be rules in place to project use data. Specifically this includes:

  1. Self published apps (only available on publisher's sites) should not be able to access the network
  2. Network published Apps (available on a decentralized publishing platform with verification of app on its own domain - Apps that are on decentralized platform but don't have a equivalent backing version should show a warning to the user as being high risk) should still not get access until they are vetted.
  3. Vetted Apps (code-reviewed by enough people who are proven to be real people) should have access.
  4. Uses should be able to judge the trustworthiness of Apps by its rating.

Once network published Apps are implemented the following initial approach can be taken:

Check Apps and UIs for presence of 'eval', http:' and 'https:' strings to prevent network quests. Also disallow 'fetch' and XMLHttpRequest strings. And, for now, disable 'href's by disallowing that string. To make up for lack of "<a href..." provide a small library allowing UIs to navigate between Apps, and clearly document this temporary limitation.

Change build structure to common dependency tracking root

Currently I have to re-compute project dependencies every time there is a new change (across devices used to work on AIRport). This slows down development (especially older boxes) and is just plain annoying. I will introduce a common dependency tracking mechanism, across all projects need to build AIRport and AIRline by placing them one directory out the source projects.

Clean up CachedQueries when Observable has no subscriptions

Right how Observable queries get added to ActiveQueries and are never cleaned up. They probably need to be cleaned out when the Observable is no longer subscribed to (and when a UI that holds the subscriptions gets unloaded/replaced by another UI). There should be a way to re-add a query to ActiveQueries in case the Observable is still around and can be re-subscribed to.

Ensure enforcement AIR entities and AIR-to-AIR relations.

Investigate if there is enforcement of AIR entities only in feature schemas. There needs to be enforcement both at build time and at run time.

All relations (in AIR entities) must point to other AIR entities (enforced at build and run time) (with only known exception being UserAccount table).

Add support for Application signatures.

Applications should have a public signing key associated with them. Application versions should be signed. Signature should be verified before the application is installed.

Implement CascadeGraphVerifier

When Apps run across network boundaries and data graphs passed into the 'save' operation are serialized:

For save operations (OperationManager.performSave), the input should be verified in its serialized form before an entity graph is reconstructed. Right now it's not an issue since objects are passed between browser and there is no need for this verification.

Lock down access to internal entities

As of right now only the following internal entities can be linked to:

@airport/travel-document-checkpoint:
UserAccount

@airport/holding-pattern:
Actor
Repository

Add checks at build (code generation) time and App initialization time to make sure that only these internal framework entities are accessed.

Investigate support for creation of multiple repositories per outermost API call

Right now AIRport limits creation of Repositories to one per outermost API call (including all calls to APIs of other Apps). This is tracked through IRootTransaction.newRepository. This leads to focused operations, where one App creates a Repository and then other Apps may may calls that add data to that Repository. However this also limits ability to create multiple related Repositories at the same time.

Note that you can still update records to other Repositories (thus moving them between Repositories) and you can add (create) records in other existing Repositories. There are no limitations on how many existing Repositories are used for those operations.

After initial internal and 3rd party Apps are written investigate if this limitation needs to be adjusted (by perhaps allowing one Repository per Dao.save call).

Implement QValidator

Need to add additional SQL (low level, internal) validation that is performed on the SQL input objects before the queries/statements are executed. The implementation is stubbed out in the QValidator object. It's methods need to be implemented.

Allow manual disabling for One side cross-repository link

By default the relation edges (many-to-many links) will be present in the repository containing the One side of a relationship. There may be cases where the developer intends to allow only one-directional repository navigation (from the Many side).

Provide a way to remove a link to one of repositories (or links to all or many of repositories) on the Many side. This can be done by creating a property on the edge table that will signify if a given relationship should be present in the One side repository or not.

Maintenance of this relationship may be done via a utility function like:

unlinkRepositories(oneSideEntity, oneSideEntity.crossRepositoryOneToMany)
unlinkRepositories(oneSideEntity, oneSideEntity.crossRepositoryOneToMany, oneSideEntity.crossRepositoryOneToMany[2])
unlinkRepositories(oneSideEntity, oneSideEntity.crossRepositoryOneToMany, oneSideEntity.crossRepositoryOneToMany.slice(2, 4))

an inverse function can also be provided:

linkRepositories(oneSideEntity, oneSideEntity.crossRepositoryOneToMany)
linkRepositories(oneSideEntity, oneSideEntity.crossRepositoryOneToMany, oneSideEntity.crossRepositoryOneToMany[2])
linkRepositories(oneSideEntity, oneSideEntity.crossRepositoryOneToMany, oneSideEntity.crossRepositoryOneToMany.slice(2, 4))

This will flip a flag on the edge record for each of the OneToMany child records specified. The in memory value of this can be kept in entityState property (which may have to be expanded from being a plain enum to a record)

Copying and maintaining of referenced records into referencing Repositories

Records that are pointed to by @ ManyToOne() relations but are in different repositories should be:

  1. Recorded in the Repository Transaction History (keeping original Actor and Actor Record Ids, changing the Repository Id)
  2. Created when the Repository is loaded
  3. Updated (with different Repository Id) the referenced repository is loaded. Also the values need to be checked (by scanning for changes in the Repository transaction history) and updated where needed.
  4. @ ManyToOne() references to these records should be updated.

This solves a number of issues:

  1. It is possible that the referenced Repository is not accessible to some of the users of the referencing repository. In this case the queries that use @ ManyToOne() relations should still return full results. This is true especially if the query results are constrained by the records pointed to by the @ ManyToOne() relations.
  2. Even if the referenced Repository is accessible the query results may not return full data (or any data) until the referenced Repository is loaded, thus causing lag.

NOTE: This will impact native Foreign Keys in the SqlJs table (currently they are not generated so not an issue).

File System integration

Preferably integrate with files.chainsafe. It is perfect for AIRport since it provides shared folders (with both view and edit access) that are limited to only the invited members.

Move Apps to QuickJs VMs

Right now Apps run in IFrames. And every iframe drags along with it a lot of framework code needed to make queries (and cross application queries) run. This won't scale once there are many AIRport apps (with maybe dozens if not hundreds loaded into memory at a time). A solution for this (and to better enforce security, which is even more important) is to extract just he App code and move it into a controlled QuickJs VM. The needed framework code will only be loaded once (with the core framework). This will also remove the need for async messaging between IFrames and for initialization of App metadata (including all referenced Apps) within the App IFrame isolates.

There result will be a more secure installation that runs with less RAM and CPU and scales much better.

Do not load Repositories if they span Apps that are not loaded.

This is a temporary solution to prevent loading issues:

If a retrieved repository contains data stored in Apps that are not loaded - do not load it. This can happen when Repositories are actively pre-fetched based on references (in previously loaded Repositories).

This covers the scenario enabled by #23

This is improved upon by #25

Investigate moving UIs into QuickJs

With initial release of AIRport UIs will run in IFrames (or optionally as their own tabs). Before their source is loaded it will be scanned for any network access attempts. It will be even more secure to run the UIs in QuickJs (with all DOM and API interaction defined explicitly). Investigate if and when to do this as well as performance impact.

Authorization system

A repository level authorization system is needed.

[Note, there is no need for user facing App authorization since all Apps are sandboxed and cannot transfer data outside of their sandbox (except to and from AIRport). Once advertisement revenue is distributed between apps there may be a need for one but it really also makes (perhaps even more sense) to let the best App/UI win (which is also mitigated by the fact that Apps should get the bulk of the revenue, vs UIs) and that they control persistence to their schemas and the business logic around that.]

Since initial integration will be with files.chainsafe.io initial roles are limited to:

  • Owner
  • Can-write
  • Can-read

Permissions are always maintained by the owner.

Thus, this defines the scope of the original implementation. This should be tracked in Repository to ensure no-vendor lock-in - Repository authorization should be portable off Files API. This is necessary especially for the case of Repository being ported to permanent storage (like ArWeave or Lighthouse).

In the future expanded features can be:

  1. The initial user is by default the administrator user for a Repository.
  2. Admin users have read and write permissions.
  3. Admin users can add and remove read and write permissions to any non-admin user of the repository.
  4. Initial user is the super admin and can grand and remove Admin status.
  5. AIRport enforces all of the above rules.

Add default LEFT JOIN support

Currently all joined entities in the FROM clause must explicitly specify the join type. In virtually all of the cases the join will be LEFT JOIN. Implement default join specification. Example, the following FROM clause definition:

FROM: [
c = Q.Conversation,
c.comments.LEFT_JOIN(),
p = c.participants.LEFT_JOIN(),
p.userAccount.LEFT_JOIN()
]

will also be expressed as:

FROM: [
c = Q.Conversation,
c.JOIN.comments,
p = c.JOIN.participants,
p.JOIN.userAccount
]

Note q.LEFT_JOIN shouldn't be an option to discourage inner joins (which break most functionality due to):

  • No parent returned from sub-trees where there there is no nested data via @OneToMany relationships
  • Repositories having dangling links (would be fixed if records are copied into repositories #22) via @manytoone relationships

Implement Observables

Observables will be important once the peer-to-peer synchronization connections exist but App implementation will start before then so they should be in place from the start.

Finish the existing partial implementation of Observables and test it.

Implement ApiValidator

All API operation Inputs should be validated against the specification on the application.json
Implement ApiValidator (currently stubbed out). Investigate if additional API input information need to be collected (and validated).

Implement OnlineManager

Right now AIRport assumes that it's always online. There should be checks to see if it has a network connection (and is able to synchronize repositories) or if the sync operations need to be delayed.

Implement OnlineManager

Provide Q member property in DAOs

Currently Q objects must be manually imported from underlying Apps. This prevents auto discovery of Entities in the same app and across all Apps imported by the project. That in turn goes against the "Developer Friendly" goal of AIRport and should be implemented soon after initial release.

Technically this will mean that at build time all included Apps will need to to be additionally pre-processed (they are already scanned) and all entity objects will need links created for their Q objects:

this.Q.MyEntity - link to a Q object that is internal to this application
this.Q.['domain-name']['app-name'].OtherAppEntity - link to a Q object in a related App.

Add SAVEPOINTs for internal APIs - enable data modifications in Observable streams

Right now the only way to make data modifications from within an observable stream handler is to setup a second application and make a call to it (this forcing a transaction creation). This is inconvenient and should be improved:

API calls made within an Application should also have SAVEPOINTs associated with them. This allows for better error handling. More importantly it this allows for transactions to be kicked off outside of an external API call context. A perfect example of this is performing a modification within an Observable stream handler.

To accomplish this every @Api() call should be instrumented with a transactional wrapper. This means that there should be code generated specifically for the @Api() decorators (and an exception needs to be added to @airport/taxiway not to strip that code). The instrumented code will:

  1. reach out to AIRport server and forward the request to the AIRport server - which will start a nested transaction
  2. wait for a response from the AIRport server and return the results

The instrumented code should only be invoked if the call wasn't made externally (there should be a flag in the @airport/tower harness to disable the instrumented code for external calls).

Only the @Api() calls that return Promises will be instrumented (can't do transactions from within calls that return Observables).

NOTE: An alternative implementation would start and stop transactions but that would double the amount of messaging required between the App Frames and the parent window (in Web Mode).

Load Repositories based on Ids in query inputs.

Pre process query inputs and check for Repository Ids in them. Check these Repository Ids against loaded Repositories. If a referenced Repository is not loaded, load it and all of the Repositories it references. If a Repository is loaded but the Repositories it depends on are not loaded, load the dependencies of those Repositories.

Also re-query on remote repository changes, based on repository Ids in the query input fields.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.