Giter Club home page Giter Club logo

polkadot-spec's Introduction

Polkadot Protocol Specification

Introduction

This repository contains the official specification for the Polkadot Protocol. The latest releases of the Polkadot Protocol Specification can be found on spec.polkadot.network or our GitHub Releases page.

Process

Everyone, but specifically implementers and members of the Technical Fellowship, are welcome to open PRs to this repo and contribute to the specification. A PR is merged as soon as two members of the Spec Committee accept the PR.

Spec Committee

The Spec Committee consists of Researchers, Researcher Engineers, and Implementers of the Polkadot Protocol that actively contribute (= opening/contributing to PRs or issues regarding the spec) to the specification. For now and compared to the Technical Fellowship, the Spec Committee's responsibility is clearly defined as maintaining the specification of the protocol. In case of inactivity for three months, members will be removed again.

Web3 Foundation

Quadrivium

Smoldot

Parity

Local Development

To clone, build and serve the website locally, run the following commands:

git clone --recurse-submodules https://github.com/w3f/polkadot-spec
cd polkadot-spec
git submodule update --remote
cd website
npm i
npm run build # or build_with_kaitai to also rebuild kaitai SVG files
npm run serve

If you already have the repo cloned, remember to update it and the submodule before making any changes:

git pull
git submodule update --remote

Because of the "complex" build, you can't see the changes in real time if you directly edit the docs folder. There are two workarounds:

  • use a Markdown plugin or editor (e.g., the extension Markdown All in One for VSCode) to see the changes in real time. You won't be able to see the placeholders rendered and other elements, but you'll be able to render the markdown and latex;

  • build the website the first time, then run npm run start instead of npm run serve, and edit the files inside the website/docs folder. This will allow you to see the changes in real time, but you have to remember to bring the changes inside the docs folder before committing. Also, in this way, you can't put the placeholders and other elements inside the markdown files of website/docs as they wouldn't be rendered.

You can find the markdown files inside the docs folder.

When building, the scripts inside preBuild will generate a docs folder, from which Docusaurus will render the website. Then, the rendered content will be modified by the plugins in the browser.

Formatting

LaTeX

You can use LaTeX inside the markdown files using the following syntax:

$ LaTeX here $

or

$$
LaTeX here 
$$

Numeration System

Inside preBuild, you can find the script numerationSystem. This will assign to several entities a number and substitute the placeholders inside the markdown files. This is done to avoid having to manually update the numbers when adding new entities.

This is the structure of the spec:

- Macro Chapter X
    1. Chapter A
        - Section 1.1
            ... subsections
        - Section 1.2
    2. Chapter B
        - Section 2.1
            ... subsections
        - Section 2.2
- Macro Chapter Y
    etc.

Example:

- Polkadot Host
    1. Overview
        1.1 Light Client
        ...
    2. State and Transitions
        ...

The entities involved are:

  • Chapters
  • Sections
  • Definitions
  • Algorithms
  • Tables
  • Images

Those defined as "Macro Chapters" will not be numbered.

Chapters

To write a new chapter, use the following syntax:

---
title: -chap-num- Chapter Title
---
<!-- Chapter content here -->

The placeholder -chap-num- will be replaced by the number assigned by numerationSystem.

If you add a chapter (or "Macro Chapter"), you must also add it to the sidebars.js file and adjust the numbers of the other chapters.

Sections

To write a new section, use the following syntax:

## -sec-num- Section name {#id-section-name}
  • Use a markdown header from H2 to H5 included, so the maximum depth is a.b.c.d.e (H2 is a.b).
  • Put the placeholder -sec-num- in the header, which will be replaced;
  • Add an id to the header, which will be used to reference the section.

Definitions

To write a definition:

###### Definition -def-num- Runtime Pointer {#defn-runtime-pointer}
  • Use a markdown H6 header (######);
  • Put the placeholder -def-num- in the header;
  • Add an id to the header.

Then, you should include the definition content inside the custom admonition :::definition (you can find all the custom admonitions inside src/theme/Admonition/Types.js).

So the final result will be the following:

###### Definition -def-num- Runtime Pointer {#defn-runtime-pointer}
:::definition <!-- Open admonition -->

Definition content here

::: <!-- Close admonition -->

Algorithms

To define an algorithm, use the same syntax as for definitions but with the placeholder -algo-num-:

###### Algorithm -algo-num- Aggregate-Key {#algo-aggregate-key}

At the top of the page, you must include the Pseudocode component and the LaTeX algorithm you want to render:

---
title: -chap-num- States and Transitions
---
import Pseudocode from '@site/src/components/Pseudocode';
import aggregateKey from '!!raw-loader!@site/src/algorithms/aggregateKey.tex';

After this, you can build the algorithm using the admonition :::algorithm, and using the Pseudocode component (refer to the file to know more about its props). This will be the final result:

---
title: -chap-num- States and Transitions
---
import Pseudocode from '@site/src/components/Pseudocode';
import aggregateKey from '!!raw-loader!@site/src/algorithms/aggregateKey.tex';

<!-- Page content here -->

###### Algorithm -algo-num- Aggregate-Key {#algo-aggregate-key}
:::algorithm
<Pseudocode
    content={aggregateKey}
    algID="aggregateKey"
    options={{ "lineNumber": true }}
/>

<!-- Algorithm description here -->
:::

Tables and Images

To define a table or an image, use the same syntax as for definitions and algorithms (always using an H6 header) but with the placeholder -tab-num- or -img-num-:

###### Table -tab-num- Name {#tab-name}

or

###### Image -img-num- Name {#img-name}

For these two entities, you won't need to use any component or admonition.

References

To reference any of the entities from anywhere in the website, you have to use the following syntax:

[Entity -xxx-num-ref-](entity-page#entity-id)
  • Use a markdown link;
  • Put the placeholder -xxx-num-ref- in the text link, which will be replaced (xxx depends on the entity, for example -def-num-ref- for definitions);
  • The link should point to the header id of the definition, with the page name as a prefix (if entity-page.md is a page, ".md" must be omitted).

Bibliography

The cited works are defined inside bibliography.bib. To cite a work, use the following syntax:

[@work-id]

Automatically, the bibliography will be generated at the end of the page.

Broken Links

During the preBuild, the external links in the markdown files will be checked.
After the build, the internal links will be checked.
If any link is broken, the console will show a warning.

Refer to checkBrokenInternalLinks/index.ts and checkBrokenExternalLinks/index.ts.

Resources

Docusaurus docs: https://docusaurus.io/docs/category/guides

Kaitai docs: https://doc.kaitai.io/user_guide.html

License

Any code in this repository is licensed under the GPL 3.0, and any documentation or specification is licensed under the CC BY-SA 2.0.

polkadot-spec's People

Contributors

0xcaso avatar alan-null avatar alistairstewart avatar bhargavbh avatar bkchr avatar burdges avatar dependabot[bot] avatar drskalman avatar drw3rk avatar eclesiomelojunior avatar fatemeshirazi avatar fgimenez avatar florianfranzen avatar gavofyork avatar haider-ali-ds avatar harrm avatar heversonbr avatar hyunggyujang avatar imod7 avatar jacogr avatar laboon avatar lamafab avatar lexnv avatar ltfschoen avatar noc2 avatar qdm12 avatar romanb avatar rphmeier avatar tjjfvi avatar tomaka avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

polkadot-spec's Issues

Circle CI does not clone the submodules

When I check the test vm the ~/implementation/rust/trie is empty when I manually run

 git submodule update --init --recursive --remote

Then I will get the submodules populated.

Spec Slot-number for given time

it seems that the implementation is using an absolute slot number.
this is not what is specified in Babe research paper and I'm not sure if it is
compatible with the median approach so this need to be clarified and updated
in the spec.

Define authority directory for each epoch

We need to spec how to retrieve the authority list at begining of each epoch
This is basically should be contained an algorithm which need to be called
at the begining of each epoch for babe. This also need to include how
to retrieve session key of each authority and how to verify them.

Make install should call cargo install for rust

The scripts for CMake rust support runs "cargo build" fine (when you write make it runs cargo build) but it doesn't support cargo install, when you issue "make install". So I had to write these unreliable lines:

https://github.com/w3f/polkadot-re-tests/blob/0137463f3224a8150b29184e56eb00308987f922/test/testers/rust-tester/CMakeLists.txt#L6

as temprory measure because I wanted all executables to go somewhere deterministic independent of the os, build, etc so I can run them by the Julia script. But the correct way of doing it is to modify the CMakeRust scripts so make install runs cargo install properly.

The original CMakeRust script is coming from this repo:

https://github.com/wayk/CMakeRust
:END:
nil:END:

Can not compile and link to Kagome scale implementation

I'm trying to add a command line helper which links to and tests kagome scale implementation. See here:
https://github.com/w3f/polkadot-re-tests/blob/f--kagome-scale/test/testers/cpp-tester/CMakeLists.txt#L1

However, I get these errors when I try to compile. Could you resolve it?

-- Boost version: 1.69.0
-- Configuring done
-- Generating done
-- Build files have been written to: ./polkadot-re-tests/build
[  0%] Built target git_submodule_init
[ 10%] Building CXX object implementations/cpp/kagome/core/scale/CMakeFiles/scale.dir/basic_stream.cpp.o
In file included from ./polkadot-re-tests/implementations/cpp/kagome/core/common/buffer.hpp:13,
                 from ./polkadot-re-tests/implementations/cpp/kagome/core/scale/basic_stream.hpp:9,
                 from ./polkadot-re-tests/implementations/cpp/kagome/core/scale/basic_stream.cpp:6:
./polkadot-re-tests/implementations/cpp/kagome/core/common/result.hpp:46:21: error: expected constructor, destructor, or type conversion before ‘;’ token
   Value(T)->Value<T>;
                     ^
...

Spec the digest in the header

  • the format, (probably optioned scaled vectors)
  • Options (seal etc)
  • We need to define seal (the last part added to the header which is not part of the signed part of the block?)
  • In particular we need to spec the details of each option like seal format etc.
  • it should also specify the format of log in the digest.

Write up the RPC standard

This can be done following the Substrate code. It should probably be a different indepenedent document.

Remove appendix E

It's always been intended that our networking protocol would change in the future. It shouldn't be specified yet.

The blobs in chain-candidates should be updated automatically

The Polkadot runtime built by:

$ git clone https://github.com/paritytech/polkadot.git
$ cd polkadot
$ git checkout v0.3
$ git pull
$ ./script/build.sh

Should be remade and pushed to the test repo everytime the runtime is get updated.

Alternatively, it can be remade, when the test are made but it is rather costy and lengthy to do so everytime and the runtime doesn't get changed often.

We also keep a copy of the runtime in wat format. So it is necessary to regenerate the wat file everytime the runtime gets updated. This can be done using wasm2wat using wabt toolkit.

Spec ext-network-state return value

This requires a deeper look since it includes datastructures from libp2p-dev.

You would need to call into libp2pThe local PeerId should be know, as well as the Multiaddress you are being reachable from the outside. You take these values and call PeerId.encode(), MultiAddress.iter().map(Encode::encode() And this stuff is put inot the OpaqueNetworkState. It is opaque, because it only has Vec and does not know of the actual structure of the types. But libp2p knows how these types look like.

pub struct OpaqueNetworkState {
	/// PeerId of the local node.
	pub peer_id: OpaquePeerId,
	/// List of addresses the node knows it can be reached as.
	pub external_addresses: Vec<OpaqueMultiaddr>,
}

Spec Babe Validation

The process by which a node should validate Babe header of a block (VRF etc)
needs to be speced.

move run

alos rename build_test.sh to build_tests.sh

Clarify how the Number of slots in an epoch is derived in Babe

for spec "Notation 4.4. We refer to the number of slots in epoch En by scn. scn is set to the result of calling runtime entry
BabeApi_slot_duration at the �beginning of each epoch.", does BabeApi_slot_duration return the number of slots in the epoch, or the length of each slot for that epoch?
(export "BabeApi_epoch" (func $BabeApi_epoch))
returns all details about the epoch.
https://github.com/paritytech/substrate/blob/master/node/runtime/src/lib.rs#L610
duration is in the number of slots

Write up maintain transaction pool

Currently at Section 3,2.3. Maintaining transaction pools is about moving transactions whose tags are satisfied to
transaction Queue and purging transactions whose life span is passed and couldn't make it to a target block.

CircleCI should install and run the nighty versoin of rust

I'm getting this in compiling parity's code:

error[E0554]: #![feature] may not be used on the stable release channel
  --> /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/hash256-std-hasher-0.12.4/src/lib.rs:16:35
   |
16 | #![cfg_attr(not(feature = "std"), feature(core_intrinsics))]
   |                                   ^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

For more information about this error, try `rustc --explain E0554`.

for which rustc says

root@54d6870151e0:~/project# rustc --explain E0554
Feature attributes are only allowed on the nightly release channel. Stable or
beta compilers will not comply.

Example of erroneous code (on a stable compiler):

#![feature(non_ascii_idents)] // error: #![feature] may not be used on the
                              //        stable release channel

If you need the feature, make sure to use a nightly release of the compiler
(but be warned that the feature may be removed or altered in the future).

Preliminary questions

I have asked about the status of the spec in w3f/polkadot-re-spec#10, presuming its at a stage where asking concrete questions is meaningful, let me know if that's not the case.

Questions

  1. Algorithm 3 appears to be broken,

    • i is never incremented?
    • pk_{N} should probably be in pk_{N_i}, otherwise the same partial key is added repeatedly?
  2. KeyEncode and StoredValue show two conflicting presentations for showing the domain and range of a function, StoredValue is by far the most readable.

  3. Def. 21 needs to be tightened up. State Storage is relied upon as a term, yet not formally defined (from what I see)

  4. Def. 23

    • uses an undefined single bar operator, is this meant to be array concatenation?
    • has an unnamed, unquantified variable i, not clear how to interpret this. Is this i related to the node traversal variable in Algorithm 3?
    • should l_{pk_N} is the length of the nibble array pk_{N}, so perhaps the naming l_Nwould be more succinct?

Readability

For readability, I would suggest

  • Always have backward-looking references for definitions, e.g. unlike Def. 25, this ensures that the conceptual apparatus is being built up in the natural reading direction. This is also the norm for most formal exposition, at least in my experience.

  • Properly formalize what state storage is in some way. Have precise statements about what it means for things to be or not be in storage, along with keys, values, etc. Then define state tries and other apparatus as representing or corresponding to some state storage instance. This makes it possible to unambiguously talk about an isomorphism or content equivalence, as is attempted in Def. 21. If not this, then at least something more robust then what is going on now. I now think I get what you are trying to do with this state storage concept, but it took too long to disambiguate.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.