sigp / lighthouse Goto Github PK
View Code? Open in Web Editor NEWEthereum consensus client in Rust
Home Page: https://lighthouse.sigmaprime.io/
License: Apache License 2.0
Ethereum consensus client in Rust
Home Page: https://lighthouse.sigmaprime.io/
License: Apache License 2.0
The ssz crate has been updated so it follows the beacon_chain version, however the README has not been updated to describe it.
Add a README.md which describes:
Encode
and Decode
traits.Shuffle needs the following work:
Implement the function described here: ethereum/consensus-specs#54
Function doesn't exist.
Function should exist.
AFAIK, the function is still experimental so be on the lookout for bugs and optimisations.
At this stage, I think we should implement it as a separate crate.
We presently have a ssz
crate which uses the wrong type of serialization. It follows the ethereum/research/py_ssz scheme instead of the ethereum/beacon_chain scheme.
Also, it doesn't decode.
It should follow the beacon_chain serialization format and support encoding and decoding.
We only need it for Blocks and AttestationRecords at the moment, so it would be ideal to prioritize them.
lighthouse/ssz/src/impl_encode.rs
Line 50 in cc2e210
Presently, SSZ prefixes a Hash32 with a length prefix. It should not do this, as a Hash32 is fixed-length and does not need a length prefix.
When this is implemented, it will break the SszBlock
struct. I am more than happy to fix that struct personally when the time comes.
The clippy linter has a lot to say about our ssz implementation. We'd like to remove these warnings so we can eventually start failing builds if they trigger clippy.
Clippy outputs warnings when run against the ssz crate.
Clippy should not have any warnings.
The crate was written during an experimental phase of SSZ and is not as elegant as we would like. We're not attached to it and would be open to suggestions on a restructure. Please make a new issue if you'd like to restructure :)
Add subtasks.
In particular this requires #11
I have created a test for BooleanBitfield that I think should pass, but fails.
Test fails
I think this test should pass
I haven't looked into how to resolve it.
It looks like this module is not being used. It does provide duplicate definitions of AttesterMap
and ProposerMap
which is confusing. It seems like this module is just dangling after a refactor.
it exists.
this module should not exist if it is not being used.
Let's just remove it.
Opening this issue to confirm my suspicion that it is just dangling after a refactor. Otherwise just close this :)
The spec has update and AttestationRecord
needs to be updated.
This will likely be a fairly significant refactor as the AttestationRecord
validation will need to be upated.
Currently validator delegation, more specifically, generate_cycle()
in transition/delegation/validator.rs
could do with extra logic in the tests.
Specifically checking expected committee sizes and more tests checking edge cases.
The genesis_states
function in the chain
crate has been stubbed out as new updates to the spec have made it mostly redundant.
Implement the new genesis code after #96 has been merged.
As detailed here: https://ethresear.ch/t/blob-serialisation/1705
The spec updated and the CrystallizedState
and ActiveState
are now merged into the one object.
We have two, distinct states.
The states should be unified
Update types::Block
to bring it inline with the latest spec updates.
Presently, we're using the block struct from several spec revisions ago.
Be up-to-date
Block
-> BeaconBlock
.SszBlock
.This issue is presently on-hold until the specification is stable.
This is a "mega issue" which contains multiple discreet tasks. Mega issues exist to avoid cluttering the issues page a multitude of small tasks.
If you wish to work on one of these tasks, comment below and a maintainer will break the task out into a separate issue.
The definition for all structs can be found in the Data Structures section of the specification.
Types are to be defined in separate files in the lighthouse/beacon_chain/types
crate here.
All of these tasks should be quite simple.
ShardAndCommittee
struct.SlashableVoteData
struct.AttestationData
struct.AttestationDataAndCustodyBit
struct.DepositInput
struct.BeaconBlock
struct.BeaconBlockBody
struct.BeaconState
struct.ValidatorRecord
struct.CrosslinkRecord
struct.ShardCommittee
struct.DepositRootVote
struct.ValidatorRegistryDeltaBlock
struct.Implement our own version of the types that are currently pulled from ethereum_types
removing the create as a dependency.
ValidatorRecord
has been outdated. Update it and all dependencies.
validator_induction is a definite dependency and it also needs to be updated to reflect the new spec definition. To do this, you'll need the presently-undefined Deposit
and DepositParameters
structs.
ValidatorRecord
struct.Deposit
and DepositParameters
structs.types::validator_registration::ValidatorRegistration
struct.This issue here details an issue with the spec:
We have implemented a method that doesn't underflow or accept blocks until the min_attestation_inclusion_delay
'th block.
We must match the spec, but the spec is wrong IMO.
Wait until this issue is resolved, then either close this issue or update our code.
Presently the code-base is missing the following features for BLS signatures and public keys:
Clone
trait)With regards to (1), I'm not even certain if aggregating the keys then verifying them even gives a better result. I.e., is it actually quicker to add them all together then verify once, or is it quicker to just verify them all individually? I'm presently assuming the latter is quicker but I don't know for sure.
My personal rustfmt
extension failed and a lot of the code I wrote is not formatted using rustfmt
. I would like to run it across the entire code base.
Code is not formatted using rustfmt
.
Code should be formatted using rustfmt
.
rustfmt
on the entire codebaseI will manage this myself, I don't need help on this one :)
Seems like we are missing a check on the incoming block data during validation.
We want to make sure that the block's ancestor_hashes
align with our expectations given our local view of the chain.
We deserialized ancestor_hashes
and just accept the data as given.
The block should be invalid if we don't derive the expected updates to the ancestor_hashes
.
We should check that every block updates the ancestor_hashes
according to spec:
copied here for convenience (double check w/ latest spec before implementing...):
"
Also, check that the block's ancestor_hashes array was correctly updated, using the following algorithm:
def update_ancestor_hashes(parent_ancestor_hashes: List[Hash32],
parent_slot_number: int,
parent_hash: Hash32) -> List[Hash32]:
new_ancestor_hashes = copy.copy(parent_ancestor_hashes)
for i in range(32):
if parent_slot_number % 2**i == 0:
new_ancestor_hashes[i] = parent_hash
return new_ancestor_hashes
"
The spec declares that BLSVerify
should take a "domain".
Our code base has no concept of a domain.
Match the spec.
I am not aware how domain is implemented, this is a placeholder issue as a TODO
was created in the code.
SSZ tree hashing is not yet implemented, therefore we have had to "stub out" some of the canonical root methods.
The following states have the canonical_root
method "stubbed out":
AttestationData
canonical_root()
should return the canonical hash of an object. In all cases so far, this should be the SSZ tree hash (aka merkle root).
canonical_root
methods on all necessary functions.The BeaconChain
struct is still using CrystallizedState
or ActiveState
.
Uses old structs.
Not use old structs.
Refactor!
There are some basic implementations of "stores" here:
https://github.com/sigp/lighthouse/tree/master/lighthouse/db/stores
The goal of the stores
concept is to abstract database operations away from "higher-level" parts of the application. Each store is initialized with some underlying database implementing the ClientDB
trait. The store provides read/write access to specific parts of the database. Stores are separated by topic (e.g., blocks, validations) for two main reasons (a) it's nice to separate things and have smaller files and, (b) having separate stores helps to communicate which parts of the database some function might access to.
In the near future, stores may implementing caching, bloom filters and other whiz-bang things to speed up data access.
The ClientDB
trait has a concept of "columns" to help separate the key space between stores. It's up to the underlying database as to how it implements these columns. MemoryDB just adds a prefix to each key in order to separate columns. DiskDB (RocksDB) might choose to implement some columns as actual RocksDB columns and others as just prefixes in the same physical column.
Presently, the tests for the stores are pretty dismal. It would be great to have some tests comprehensive tests. We only need to test against MemoryDB, using DiskDB is cumbersome and the two should always perform identically -- if they don't it's an issue for the databases ClientDB implementation.
Happy to assist anyone who wants to work on this :)
Implement a test suite for the BeaconChain
struct.
Presently there are no end-to-end tests for the BeaconChain
struct. It's very important that this piece is heavily tested.
BeaconBlock
has been split into BeaconBlock
and BeaconBlockBody
.
The code base must be updated to suit.
Implement the add validator routine as defined: https://github.com/ethereum/eth2.0-specs/blob/master/specs/casper_sharding_v2.1.md#routine-for-adding-a-validator
We have abstracted the database into a ClientDB
trait and implemented a DiskDB
using RocksDB for production.
It would be useful to have an implementation of an in-memory database that can be used in testing. The benefits would be speed (i.e., RAM is faster than disk) and convenience (we don't need to touch the filesystem).
The in-memory database doesn't need to be particularly optimized or elegant, just the fact it's in memory should make it fast enough. I would suggest a two-layer nested hashmap to simulate columns. E.g., HashMap<HashMap<Vec<u8>>>
. It will need to implement some Mutex/RwLock internally so it can be sent around threads like RocksDB can be.
It should live in a memory_db.rs
file here and I think the struct should be called MemoryDB
.
TODO: fill out this description
Significant spec updates have required a re-write of block processing.
All existing block processing code has been removed and stubbed out.
Re-implement block processing
The shuffle()
function in the Eth 2.0 spec has changed. We need to update our function.
Resources:
RAND_MAX
is presently set to 2**24
.
RAND_MAX
should be 2**24 - 1
.
RAND_MAX
Hint: shuffling_sandbox cmd: $ python sandbox.py print --list-size LIST_SIZE
Implement a generic "wrapper" trait so we can use a proper, persistent db (e.g., Rocks) during prod and a fast, transient db (e.g., memory) during testing.
There is an implementation of a Boolean Bitfield here:
https://github.com/sigp/lighthouse/tree/master/boolean-bitfield
It (kinda) does the job for now, but it really needs some work done. If you spend some time looking at it I think you'll soon find out what I mean. As an example;
usize
, however there can theoretically be usize
number of bytes meaning we can have 8 * usize
bits.On top of these two points, there's likely many chances for optimization.
get(n: usize) -> Result<bool, Error>
Get value at index n
.
Error if bit out-of-bounds (OOB) of underlying bytes.
set(n: usize, val: bool) -> Result<(bool, Error>
Set bit at index n
. Returns the previous value if successful.
Error if bit is OOB of underlying bytes.
highest_set_bit() -> Option<usize>
Returns the index of the highest set bit. Some(n)
if a bit set set, None
otherwise.
Note: this is useful because we need to reject messages if an unnecessary bit is set (e.g. if there are 10 voters and the 11th bit is set
num_bytes() -> usize
Returns the length of the underlying bytes.
_Note: useful to reject bitfields that are larger than required (e.g., if there are eight voters and two bytes -- only one byte is necessary). _
num_set_bits() -> usize
Returns the total number of set bits (i.e., how many peeps voted).
Note: I'm not 100% sure we'll use this but I suspect we will.
It would be great to have a guide that directs users on how to install Rust and run the test suite. For example:
stable
cargo test --all
)I recommend giving high-level steps that an experienced developer could follow, then linking to more detailed articles that provide "hand holding".
lighthouse/src/state/transition/epoch.rs
Line 116 in 0ae97ee
Randao for new epoch is not correctly generated. Implement the same (incomplete) method used in the reference impl:
The on-boarding docs should be updated as we learn from gaining new contributors.
This issue can serve as a list of TODO
s for items to add to the onboarding docs:
TODO
must be accompanied by a link to a Github issue. This is to assist with project management.panic
.///
to generate doc comments.What's the only thing that changes more than Australian prime ministers?
....
The Eth 2.0 spec!!! lololol
The BeaconState
object has changed and we need to update!
Items in-scope of this task:
Items specifically out-of-scope:
There are scenarios in attestation_validation::validate_attestation_signature
that are untested.
You can find the crate at /beacon_chain/attestation_validation
(once #85 has been merged into master).
Here is a non-comprehensive list to get you started:
validate_attestation_signature
function (within reason -- ask here for canonical definition of "reason")Underflow impossible due to if list.is_empty()
check above. Remove saturating_sub()
.
Description here:
lighthouse/beacon_chain/utils/ssz_helpers/src/ssz_block.rs
Lines 52 to 59 in 817b52f
This concern shouldn't be enforced at this stage, SszBlock should be general enough to handle zero attestation records.
Remove the requirement that the SSZ must be long enough to accommodate one attestation record.
Presently we do not serialize the aggregate_sig
field on an AttestationRecord
.
We need to:
aggregate_sig
on the AttestationRecord
struct.A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.