Tasks:

• Create listener for events emitted on open/new tab creation
• Create listener for events emitted on change of tab contents (add/remove uApp, network/uApp status)
• Create listener for events emitted on destroy/close tab
• Create API call integration for the state of uApps/tabs

Deploy any subproject and have a landing page in the root to show a "catalogue" of the apps

• Reorganise/check current yarn scripts

In the repository root package.json:

When triggering deploys from the root of the folder you're unlikely to only want to deploy to ipfs or gh-pages. so always deploy both for a project. Ie. delegate to yarn deploy in that project's folder

A one-click deploy script to deploy everything isn't sensible - it'd be easy to accidentally deploy more than you intended and you're more likely thinking about deploying a single project . If you want to deploy everything it's not that much more effort when deploying to do:

1. yarn deploy:landing
2. yarn deploy:burnr
3. yarn deploy:smoldot-browser-demo
4. ...

If you're adding a new project then you do:

1. Add new project in projects folder
2. Add new project to landing page
3. yarn deploy:landing
4. yarn deploy<news-project>

In the child projects package.json:

Separate scripts - in case you need to solve a broken deployment to only one of the targets (ipfs/gh-pages). I.e.

yarn deploy - deploys to both and is called by the root package.json
yarn deploy:ipfs
yarn deploy:gh-pages

• Add landing page

I suggest that a landing page should be a special type of "project" to make the mechanics of the shared deployment scripts simple.

It's easier not to deploy the landing page to IPFS initially and tackle that as a separate issue if we think we want to. Beacuse if we deploy an app the link on the landing page stays the same but it does not for IPFS (because it has a different hash). We can solve that by using IPNS but that's quite a bit more work and probably overkill at this early stage.

I.e. put the landing page in projects/landing

• Change deploy scripts to work for any project in the project folder

Yarn scripts will rely on consistent use of yarn build in subprojects to produce a deployable folder in dist in that project (as is currently the case for burnr and smoldot-browser-demo).

Special case for landing page - does not go into a subfolder in the resulting deployment

• Change deploy script to deploy into subfolders on gh-pages
• Change ipfs deploy script to update hash links in landing page

This error Boundary should replace the loader with the error message that appeared and probably a "what to do next - next steps" message.

Right now the NodeSelector is part of the Head component which is only getting rendered, when the api is initialized and connected.
The NodeSelector should always be getting rendered so that the user can choose a different provider if the connection is failing. It still needs to be aware of the api's connection status to be able to show that as the colored dot connection indicator:

When trying to run the browser demo following error appears:

index.js:93 Uncaught Error: panicked at 'called `Result::unwrap()` on an `Err` value: Error { cause: Some(Error { cause: None, desc: "Not enough data to fill buffer" }), desc: "Could not decode `AuthoritySet::authority_set_changes`" }', src/chain_spec/light_sync_state.rs:45:18
    at throw (index.js:93)
    at :1234/<anonymous>:wasm-function[376]:0x66b6f
    at :1234/<anonymous>:wasm-function[103]:0xbeaf
    at :1234/<anonymous>:wasm-function[2877]:0x23eb97
    at :1234/<anonymous>:wasm-function[2868]:0x23e58f
    at :1234/<anonymous>:wasm-function[2867]:0x23e558
    at :1234/<anonymous>:wasm-function[2876]:0x23eab1
    at :1234/<anonymous>:wasm-function[2944]:0x24560e
    at :1234/<anonymous>:wasm-function[2967]:0x2491e4
    at :1234/<anonymous>:wasm-function[1226]:0x10129c

Update to Polkadot 0.8.25

See #93 for the specification. Create the API that the UI will interact with for getting the shared state.

Issue appears with:

Uncaught TypeError: "list" argument must be an Array of Buffers
    at Function.concat (index.js:399)
    at pbkdf2Sync (pbkdf2.js:18)
    at pbkdf2Encode (encode.js:21)
    at mnemonicToMiniSecret (toMiniSecret.js:26)
    at Keyring.createFromUri (keyring.js:236)
    at Keyring.addFromUri (keyring.js:182)
    at createLocalStorageAccount (utils.ts:35)
    at App.tsx:38
    at invokePassiveEffectCreate (react-dom.development.js:23487)
    at HTMLUnknownElement.callCallback (react-dom.development.js:3945)

Similar to https://polkadot.js.org/apps/#/js - a REPL with the Api preconfigured with the SmoldotProvider on westend. This would be useful for trying things out and to demonstrate the light client.

Email subject should be corrected from Failed to Success when PR runs

• On amount input in Receive tab, generate QR to sign

Migrate Burnr from Webpack to Parcel:

• strip down dependencies
• Make MUI work in new setup
• ...

The extension-side of the bridge between the SubstrateConnectProvider and the extension. It should

• expose an APi for ending and receiving JSON RPC messages.
• establish a port for communicating with the background script

As suggested by @Stefie in #92

I would rename the project to something other than smoldot-extension to not start the next batch of confusion, though.
It's a substrate-connect extension, smoldot delivers the Wasm Light Client, but the extension should behave the same no matter if the Light client is generated by smoldot or maybe Substrate or Polkadot one day.
The smoldot-provider makes 100% sense since that's a wrapper especially for smoldot, bur for all demo apps it doesn't matter where the Light Client is coming from, therefore I'd not name applications after it.

Add testing dependencies (ts-jest + react testing library) for Smoldot Extension

if we can listen to them -> that might not be the case with SmoldotProvider - need to make sure before implementing the issue

Make the WasmProvider class generic and ass it to the Pokladot JS rpc-provider packagae for better reusability.
https://github.com/polkadot-js/api/tree/master/packages/rpc-provider

• re-use 12 or 24-worn mnemonic seed on all chains
• do not allow account imports (it's a throw-away account)
• warn when burning keys from account with funds
• ...

A problem to think about is what to do about API version mismatches regarding the extension as user installed extensions will be longlived.

Questions to answer?

• How do we deal with upgrading the API package in the extension?
• How do we manage the fact that different pages might have different versions of the API. etc etc?
• What are the strategies for dealing with a version mismatch when a page detects a different API version in the extension from the one loaded on the page?
• Does the extension need to expose to pages what API version it is running?
• When upgrading the extension with a new API version should it have a record of which pages have used it and warn the user?

To research

• Changelog in the API - see what things do actually change and what the impact is

Related to #82

If you've already loaded the westend.json and its in your cache the browser will return the cached version.

Consumers of the SmoldotProvider have no way of knowing that we failed to start syncing from bootnodes because eg. we are offline. We need to simulate as closely as sensible what the WsProvider does in this case so that the consumer can chose to show an offline UI.

These connected disconnected and error events are propagated to the Api instance so we want the following to work as expected:

const provider = new SmoldotProvider(chainSpec());
const api = await ApiPromise.create({ provider });
api.on('error', () => /* do something */);
api.on('connected', () => /* do something */);
api.on('disconnected', () => /* do something */);

Behaviour with WsProvider

Consumers of WsProvider listen to the 3 events: error, connected and disconnected :

• If WsProvider fails to connect to its remote node it only emits error. Consumers can optionally connectWithRetry
• If it connects without error it emits connected.
• If later the socket is closed it emits disconnected.

(It also has logic for trying other remote nodes but that's not relevant for a smoldot client.)

Approach

Set an interval pinger after instantiating the smoldot client to send an RPC system_health message to check the peers count.

Desired behaviour

• Don't emit connected until we see peers > 0 from system_health pinger

• Emit disconnected if peer count from system_health drops back to zero after having previously been > 0

• Emit error if the system_health request responds with peers == 0 right after after instantiating the smoldot client (analagous to error connecting on startup)

• Emit error if the system_health request responds with an error.

• Do something similar to connectWithRetry on the WsProvider https://github.com/polkadot-js/api/blob/master/packages/rpc-provider/src/ws/index.ts#L188-L201 but rather than acutally reconnecting we delay reporting the error event until a threshold time period is elapsed.

Aside

The Api has its own pinger for system_health used to keep the websocket alive. Unfortunately we cant get at the responses it receives from the provider so we have to redundantly have a second pinger in the provider.

See paritytech/smoldot#435 for a longer conversation.

• Add docker-compose files to create local networks to speed up development

Tasks

• Upstream the typescript typings to smoldot
• Extract the smoldot client test helpers into a separate package
• Implement request and subscription proxying
• Capture the tabId and URL when a Uapp connects
• Implement loading and saving database
• Add AppMediator tests
• Add ConnectionManager tests
• Clean up subscriptions on disconnect

What should the SmoldotClientManager do

The SmoldotClientManager inside the extension must broker RPC requests and responses from multiple clients residing on browser pages (aka content pages) to multiple smoldot blockchain clients in the extension (for now we have one but there will be many). This entails:

• The SmoldotProvider in each webpage will issue RPC requests with ids in its own namespace (starting from 1). The SmoldotClientManager must maintain a translation table of client request ids to the actual request ids and send the correct responses to the correct clients. The translation table will be a map of clients to ID mappings where an ID mapping is the originating request id and the actual request id sent to smoldot. The SmoldotClientManager will change the request and response IDs in the JSON on the fly and maintain the list of live mappings.
  • Remove the ID mapping when the response has been received for non-subscriptions
  • Remove the ID mapping when an unsubscribe request / response has happened
  • Remove all mappings when a page disconnects from the extension
• The SmoldotClientManager must detect when clients disconnect and send unsubscribe messages for all that clients' subscriptions.
• The SmoldotClientManager could also intelligently cache queries in the future (out of scope for now).

Description

Users have to take more steps than necessary and have a confusing first run experience when trying out the smoldot browser demo.

What should happen?

A user should be able to perform the following steps:

git clone https://github.com/paritytech/substrate-connect
cd substrate-connect
yarn install
cd projects/smoldot-browser-demo
yarn run dev
xdg-open https://localhost:1234/

What actually happens

Cloning and running yarn install works fine, but there are a bunch of peer dependency warnings:

warning "workspace-aggregator-e155ce1d-c4fb-4b15-87b1-240ffacf2c35 > @substrate/burnr > @polkadot/[email protected]" has incorrect peer dependency "@polkadot/[email protected]".
warning "workspace-aggregator-e155ce1d-c4fb-4b15-87b1-240ffacf2c35 > @substrate/burnr > @polkadot/[email protected]" has incorrect peer dependency "@polkadot/[email protected]".
warning "workspace-aggregator-e155ce1d-c4fb-4b15-87b1-240ffacf2c35 > @substrate/burnr > @polkadot/[email protected]" has unmet peer dependency "react-is@*".
warning "workspace-aggregator-e155ce1d-c4fb-4b15-87b1-240ffacf2c35 > @substrate/burnr > [email protected]" has unmet peer dependency "react-is@>= 16.8.0".
warning "workspace-aggregator-e155ce1d-c4fb-4b15-87b1-240ffacf2c35 > @substrate/burnr > @typescript-eslint/[email protected]" has incorrect peer dependency "eslint@^5.0.0 || ^6.0.0".
warning "workspace-aggregator-e155ce1d-c4fb-4b15-87b1-240ffacf2c35 > @substrate/burnr > @typescript-eslint/[email protected]" has incorrect peer dependency "eslint@^5.0.0 || ^6.0.0".
warning "workspace-aggregator-e155ce1d-c4fb-4b15-87b1-240ffacf2c35 > @substrate/burnr > [email protected]" has incorrect peer dependency "jest@>=25 <26".
warning "workspace-aggregator-e155ce1d-c4fb-4b15-87b1-240ffacf2c35 > @substrate/burnr > @polkadot/react-identicon > @polkadot/[email protected]" has unmet peer dependency "@polkadot/networks@*".
warning "workspace-aggregator-e155ce1d-c4fb-4b15-87b1-240ffacf2c35 > @substrate/burnr > @polkadot/dev > [email protected]" has incorrect peer dependency "typedoc@>=0.20.0".

• yarn run dev fails becuase smoldot-provider has not been built

Attempting to remedy by doing:

cd ../../packages/smoldot-provider
yarn build

Fails with a lot of typescript errors about react enzyme and webpack dev server (none of which are used by this package). Running yarn build a second time builds successfully.

• 1 common component for each network (configuration file maybe?)
• 1 component for each open tab
  

Introduction

Create an end-to-end prototype to get more insight into this:

@gavofyork:matrix.parity.io
so whether it's substrate-lite or substrate --light, i don't really care, but what it should be doing is running either:
in-browser (in a site's instance, this should allow completely new users to use the site without needing to go to (RPC) servers)
in-browser-extension (better, since it's stateful between restarts and across sites)

Theory

We can make a polkadot-js API provider that makes the use of an extension transparent to the application developer.

Proposed API :

import { ApiPromise } from '@polkadot/api';
import { SubtrateConnectProvider } from './';  // name up to debate :)

const provider = new SubstrateConnectProvider('westend');
await provider.connect();
const api = await ApiPromise.create({ provider });

Proposed flow:

Extension installed: Web page -> SmoldotConnectProvider -> extension -> SmoldotClientManager -> smoldot
Extension not installed: Webpage -> SmoldotConnectProvider -> SmoldotProvider -> smoldot

Aim

• Understand better the integation points
• Validate that this is possible.
• Get a feel for the proposed API. Is it too magic? Should we be more explicit about the detection?

Approach

Create a browser extension that is not configurable and has minimal UI that is used transparently by webpages instead of embedding a node when the extension is installed.

• Create an extension that runs a single Smoldot browser node (westend) - using the existing SmoldotProvider and PolkadotJS API.
• Create limited UI to show basic node status.
• The limited UI should follow the design guidelines in Figma
• The extension should expose an API for webpages to interact with the node.
• The API should give the impression that their is more than one possible node to interact with. I.e. JavaScript running on a page must specifiy the name of the node they intend to interact with.
• Create a wrapper provider that detects the extension and uses its API if it is installed otherwise falls back to using the existing SmoldotProvider which embeds a smoldot client in the current page.

What should the SmoldotClientManager do

The SmoldotClientManager inside the extension must broker RPC requests and responses from multiple clients residing on browser pages (aka content pages) to multiple smoldot blockchain clients in the extension (for now we have one but there will be many). This entails:

• The SmoldotProvider in each webpage will issue RPC requests with ids in its own namespace (starting from 1). The SmoldotClientManager must maintain a translation table of client request ids to the actual request ids and send the correct responses to the correct clients. The translation table will be a map of clients to ID mappings where an ID mapping is the originating request id and the actual request id sent to smoldot. The SmoldotClientManager will change the request and response IDs in the JSON on the fly and maintain the list of live mappings.
  • Remove the ID mapping when the response has been received for non-subscriptions
  • Remove the ID mapping when an unsubscribe request / response has happened
  • Remove all mappings when a page disconnects from the extension
• The SmoldotClientManager must detect when clients disconnect and send unsubscribe messages for all that clients' subscriptions.
• The SmoldotClientManager could also intelligently cache queries in the future (out of scope for now).

With gzip or brotli

This creates an error upon yarn.

Steps to reproduce:

• yarn run clean
• yarn

Assuming we're using Material UI, #100 and #92 need theme component to rely on

• search bar and "add chainspec" button
• component for local/known networks
• component for unknown networks
• common popup up for each network for updating chain specs

Every Balance, Block number etc should be declared as the corresponding types from Polkador JS types, not as number, because number is a 64 bit in JavaScript, but many integers in Polkadot/ Substrate can be larger than that (u128 or u256)

import { Balance } from '@polkadot/types/interfaces';

export interface MyType {
	free: Balance;
	reserved: Balance;
	feeFrozen: Balance;
}

-> https://www.npmjs.com/package/store
-> Web Storage API / IndexedDB
-> LocalStorage / SessionStorage

Introduction

Create an end-to-end prototype to get more insight into this:

@gavofyork:matrix.parity.io
so whether it's substrate-lite or substrate --light, i don't really care, but what it should be doing is running either:
in-browser (in a site's instance, this should allow completely new users to use the site without needing to go to (RPC) servers)
in-browser-extension (better, since it's stateful between restarts and across sites)

This is an umbrella ticket to describe the architecture and design needed to have a working end-to-end prototype. It is a migration from #82

All tasks are tracked under the v0 Milestone

Theory

We can make a polkadot-js API provider that makes the use of an extension transparent to the application developer.

Proposed API for using the extension from a web page

A page exists in a browser tab and a page may instantiate multiple UApps to talk to the extension. Each UApp may talk to multiple networks (blockchains).

Proposed flow:

Substrate-connect tries to detect the availability of extension on browser that uApp runs on:
Extension installed: Web page -> sc.detect -> ExtensionProvider (extension) -> smoldot
Extension not installed: Webpage -> sc.detect -> SmoldotProvider -> smoldot

Proposed developer experience - public API for substrate connect

import { UApp } from '@substrate/connect';

// Create a new UApp with a unique name
const app = new UApp('burnr-wallet');
const westend = app.detect('westend'); 
const kusama = app.detect('kusama'); 

const chain = await westend.rpc.system.chain();
westend.rpc.chain.subscribeNewHeads((lastHeader) => {
  console.log(`${chain}: last block #${lastHeader.number} has hash ${lastHeader.hash}`);
});

const ksmChain = await kusama.rpc.system.chain();
kusama.rpc.chain.subscribeNewHeads((lastHeader) => {
  console.log(`${ksmChain}: last block #${lastHeader.number} has hash ${lastHeader.hash}`);
});

// etc ...

westend.disconnect();
kusama.disconnect();

// Create another UApp
const sbd = new UApp('smoldot-browser-demo');
const sbdWestend = sbd.detect('westend'); 

// interact with chain

sbdWestend.disconnect();

The UI

Designs for the extension UI

How will the extension UI integrate with the connection management

The ClientManager of the extension is primarily concerned with managing communication ports from UApps and the smoldot clients and brokers communication between them.

In contrast the UI is primarily concerned with what apps are on which browser tab (and URL) and which networks those apps connect to.

To achieve this the background will also maintain and expose a shared state data structure as a tree that groups the apps under the browser tab where they were instantiated, the URL of that tab and their connections to networks . It will also be responsible for triggering events when the state associated with a tab changes.

How will the UI interact with the state?

const manager = chrome.runtime.getBackgroundPage().manager;
const state = manager.getState();

manager.on('stateChanged', () => {
  const newState = manager.getState();
  // update the UI with the new state
});

What is the shape of the shared datastructure

The UI should not try to update the datastructure - the background will always return a new representation when asked for one (frozen, immutable?)

[{
  tabId: 0,
  name: 'burnr-wallet',  // unique name supplied by developer
  networks: [{name: 'westend'}],
},
{
  tabId: 1,
  name: 'smoldot-browser-demo',  // unique name supplied by developer
  networks: [
    {name: 'westend'},
    {name: 'kusama'}
  ],
}]

Task tracking

All tasks are tracked under the v0 milestone

The non UI extension tasks are labelled extension-bg

The UI extension tasks are labelled extension-ui

Helper to detect whether the extension is installed. It will return a SubstrateConnectProvider which is ready to connect to the extension if so otherwise it will return a SmoldotProvider (in-pade light client).

Open issue in https://github.com/paritytech/ci_cd/issues?page=2&q=is%3Aissue+is%3Aopen

The licenses are a different and inconcsistent even in the same context:

• The root license file is GPL3 but the root package.json says Apache-2.0
• The newly added projects / packages are Apache2. (I just copied the Burnr license, but is that right?)
• Smoldot is GPL3.

Which is the correct one to use? Is it the same for all or different? (categorised below to make it easier to comment)

• PolkadotJS API provider
• Browser extension
• Product demos: burnr and smoldot-browser-demo

Figure out why ts-jest doesn't work when using import.meta.url

New provider based on the SmoldotProvider but should not do healthcheck pinging and has different lifecycle events (To be decided).