near / nearcore Goto Github PK

If shard chain is taken over (>2/3 of validators controlled by an adversary), the shard chain can produce invalid state transition. Through receipts, this transition may be acted upon in the other shards.

Add code coverage tool via https://codecov.io and a shield into README

Need to figure out a way to create account outside genesis. @ilblackdragon

we can have more ergonomic API endpoints, right now, we are basically jamming everything under a transaction

Ref #7 : build unit tests for tradeoff cases of the fork choice rule.
Make a decision for how our fork choice rule will work in this cases.

Example test cases can be found in Vitalik's post

Protocol for communicating between nodes (where P2P is abstracted out)

• Epoch, endorsement, approval computation
• Fetching
• Handling of missing messages
• Message and transaction bundling

Use hash of a string as AccountId.

Current gossip is not optimized, specifically the fetching part and the rate at which we gossip.

• When one witness fetches DAG from another witness it does it in a BFS fashion one layer of missing messages at a time. We might want to optimistically ask several layers of messages at a time. This would require some heuristics.

• Currently when witness A receives an unsolicited gossip from B containing message X, if message's parents are all known to A it will add X to its DAG and send an gossip reply back to B (which helps with congestions). However, if X has missing parents A will reply to B later after all parents are known. However if while A waits for parents some other witness C gossips the same message to A, A will not reply to B anymore, and will only reply to C. This should be done better somehow.

• TxFlowTask should be able to detect when someone repeatedly spams it. This however is not a malicious behavior since it cannot be proved to be malicious so it cannot be addressed with MisbehaviorReporter, see: #118
  It should be solved with some struct we could call "SuspiciousBehaviorReporter" which we would report to and which our network layer would use to determine whether we want to bounce the messages.

Passes on local machine: https://www.dropbox.com/s/6ezwwxftz8w85co/Screenshot%202018-11-23%2009.53.06.png

but fails on the CI: https://gitlab.com/nearprotocol/nearcore/-/jobs/124255464

We should define the set of messages we want to send through and receive from network. Maybe we should have a spec for this.

• WebAssembly
• Interaction with state
• Transaction format for running functions
• Example of payment contract

we should deliberately upgrade downstream git deps

using this as a parent issue to keep track of places where we have code that should live in a test-utils module.

• primitives::types::SignedTransaction.empty
• primitives::signature::default_signature

i don't know if we are always gonna line up with rustfmt rfc's and it would be good to have an option to define our own style but also get a formatting tool.

options:

• fork rustfmt
• use non-rust specific formatter that you can tune well

this issue will also be used for discussions about style preferences we have that don't align with our current configuration.

In order to be more aligned with the interface of libp2p and other Rust libraries that use futures we need to replace some of our state-sharing concurrency with message-passing concurrency. Which would also allow us to get rid of some thread-safety primitives and hopefully be faster.

The following things should be run as tasks:

1. network_listener. Listens to the stream of network messages from libp2p and spawns network_messages_handler's;
2. network_messages_handler (currently called Protocol). We spawn a new instance of handler for every new network message received just shown in the last example here: https://tokio.rs/docs/getting-started/echo/ so that we can benefit from the concurrency.
3. TxFlowTask is as designed should run as a task.
4. A pool of Runtimes that execute the contracts.
5. RPC server.

• CLI would be on the main thread, and spawn tasks 1, 3, 4, 5, and provide them with the channels that they would use to communicate.

i think this has been cargo culted from bitcoin (where all records on each block are financial transactions).. and i think it becomes confusing when we can define transaction as the following:

1. data that is being written to blocks (sending money, contract deployment, creating an account, etc.)
2. financial transaction
3. database transaction/commit

Let's go over this document and make sure we are on the same page here:
https://www.overleaf.com/read/schffmbrbynq

Added @mikhailOK because he was also interested in solidifying these rules.

and Runtime.genesis_state should also use ChainSpec

Looks like some linker or cargo issue

https://gitlab.com/nearprotocol/nearcore/-/jobs/125040565

Most probably one that ethereum/parity uses.
Options:

• Parity: https://github.com/rust-bitcoin/rust-secp256k1
• Solana: https://github.com/briansmith/ring, e.g. use ring::signature::Ed25519KeyPair;
• Exonum: https://github.com/exonum/exonum_sodiumoxide

As I was committing some of the changes to txflow crate, my commit was blocked by the following clippy find in a crate that I synced from the master https://www.dropbox.com/s/zjbr8vrww5bukhf/Screenshot%202018-11-25%2022.02.01.png?dl=0
It is weird that clippy did not block this code when it was initially merged into master.

right now, we do some pretty arbitrary branching logic when we apply a transaction. is there a better way to represent transactions so that we can represent them as different types throughout the code base in order to make it easier to reason about what is going on?

@ilblackdragon @bowenwang1996

There are three generated tests failing. Here are the tests and the reasons:

bob_fork

Alice approves two messages from Bob, neither shall be considered an endorsements. Carol and Dave only approve one, both such approvals shall be endorsements.

bob_second_msg_in_kickout_epoch

Bob has a message that approves his own kickout message in the same epoch. Rust implementation mistakenly marks such message as kickout as well.

generated_kickouts_and_epoch_blocks_1

For some reason Carol's message is not a representative, even though it approves the previous representative, and its own kickout message has sufficient number of endorsements.

Here we keep track of the things in our current code base that reimplement the already existing functionality of the Rust futures crate. Most of them will go away once we close: #121

• tx_pool and import_queue in Client will become not needed once transactions, blocks and other things are going to be communicated using channels between the tasks, because channel is a FIFO queue itself, see: https://doc.rust-lang.org/std/sync/mpsc/

• The handshaking_peers in Protocol reimplements Future as it waits on a certain task to complete by re-implementing the polling mechanism of the futures using maintain_peers that is being regularly called on a timer, see generate_service_task.

• Currently block_production in Protocol does not reimplement the polling mechanism, because it simply calls prod_block of the Protocol which immediately provides a fake block. However, if it was not fake, it would need to wait for a consensus to finish, which would turn into the same re-implementation of the polling mechanism as discussed above.

We shouldn't wait 20 minutes for CI to finish. It seems that the cache is not working properly.

• Implement shard chain
• Sending receipts across shards

right now it takes Client which is bounded to BeaconBlock .. and depends on beacon block specifics

For MVB we use beacon chain to keep state:

• Implement blockchain tracking (storing into DB blocks and headers)
• Block production
• Block validation
• Network catch up
• Witness staking, selection and accounting for block production

Current logic doesn't correctly handle deploy method when receiver already exists.

Print the reasoning for the declined transactions in DevNet mode.

Example of errors:

• No sender account
• Error in wasm

If each account can have a custom smart contract to control the access to funds (multi sign, rules, etc), how can we ensure that the sent money came from a "valid" contract?

Few ideas:

• Amount is outside of the "account" contract
• send/deposit functions are standard and non-overridable.
• have library contract Account that others must inherit to be trusted. This implements standard functions and provides interface for extension.

Ref #5 story issue.

There is a tricky case failing, and as far as I see can be unrelated to tests tracked on #99

generated_endorse_kickout_fork

Bob create two messages (a fork) B1 and B2.

• B1 approve representative message from Alice A in epoch 0 and becomes representative message in epoch 1.
• B2 don't approve message A but is in epoch 1 so its a kickout message.

Further messages approving both B1 and B2 must discard B2 kickout message and endorse representative B1.

should probably be moved out of the network and it definitely needs to be renamed, because it is prohibitively confusing to implement Client for Client, and actually causes conflicts on IDE understanding what is going on when there are naming conflicts when they are both supposed to be in scope.

it seems like this is a trait for a blockchain, that network should rely on.

cc: @bowenwang1996 @ilblackdragon

There are two candidates currently:

• devp2p (used by Ethereum)
• libp2p (used by IPFS and Polkadot)

Implement way of specifying the chain:

• genesis state -> map of AccountId -> DBValue or Runtime::Account?
• for now -> list of public keys who are first authorities
• genesis beacon block (can be constructed automatically, not sure if we need to specify)
• any other parameters we need

• Transaction
• EpochBlock: header + body
• SnapshotBlock: header + body

With supporting signing / verifying signatures etc

There's a commented out assert in generated_kickouts in generated_tests.rs.

The test has JSON in it that can be used to visualize the graph in the visualizer that is under test-utils/txflow-test-gen.

The current version of TxFlow DAG works, but was implemented using inefficient data structures in an inefficient way. The following optimizations could improve its speed and memory usage by several times or even an order of magnitude.

• Replace u64 with more efficient and appropriate types. Current TxFlow DAG structs Message, Group, GroupApproval, GroupsPerEpoch, GroupApprovalPerEpoch, use u64 in most cases, because it was easy to write fast, this should be replaced. Note that most of these fields are only for internal usage and not going to be send across the network (except maybe epoch which is the part of the message) so it might make sense to use u8 for memory reasons, but it might not be faster CPU-wise with than u32.

• Message, Group, GroupApproval, GroupsPerEpoch, GroupApprovalPerEpoch are currently returning entire datastructures in their methods, which of course creates unnecessary allocation and deallocation. Instead they should be returning iterators wherever it is possible;

• Replace HashMap/HashSet usages, because we don't really need their flexibility of adding/removing elements at any time. Most of our mutations of these containers are limited to the construction time and therefore they can be replaced with simpler but more cost-efficient containers. Also some of our containers have limited number of elements so we might be able to use some bitmasks.

• Keeping track only of the relevant epochs. The tracking containers inside Message struct, like approved_epochs, approved_complete_epochs, etc. keep track of all past messages. It makes sense to not keep track of very old messages. This issue might become irrelevant once we implement pruning.

• Pruning. See separate issue: #116

• Implement storing state / and APIs to modify it
• Support merkle root update
• Persistent on disk (local db?)

Can use Parity for reference

• Currently TxFlow does not output consensus. They are partially computed inside TxFlow DAG but not exposed to the outside. We should compute them and stream them in the output stream of TxFlowTask.

• MisbehaviorVerifier. TxFlow should should be able to report misbehavior as soon as it detects it, which can happen in the different parts of our infrastructure: in Message, DAG, TxFlowTask structs. For now MisbehaviorVerifier does not need to send it for slashing but can simply print it or write to a file for testing.

• Replace hash with real crypto hash. We should be using crypto hash instead of the hashmap hash is some places.

• Replace signature with real signature. TxFlow might need to link to the component that does the signing, since TxFlow should not have access to private keys.

Implement data structure for sending cross shard receipts of execution

• NodeTask has the ability to start, wait, and stop.
• Node should be able to track the epoch blocks and schedule the NodeTasks.
• Node should have a mechanism to cancel NodeTasks.
• Node should be able to schedule NodeTasks.
• NodeTask should allow communication between TxFlow and Verifiers.
• NodeTask and TxFlow should handle the catch-up.
• NodeTask should have some logic for when it is lagging way behind.

In the current implementation, once TxFlow message is constructed and added to DAG it is never modified, which allows safety and efficiency. However, each TxFlow message carries heavy "aggregators" -- containers that aggregate information from the parents of the message -- which we use to compute properties of the message in near-constant time without traversing the DAG, see approved_* and computed_* fields in message/mod.rs. Since these containers grow with the number of past messages the current memory usage of the DAG grows quadratically. We can solve it by pruning the aggregators of the very old messages in the DAG. It requires several changes:

• The aggregators should be wrapped in RefCell. This will allow to have a mutable access to them even for the messages that are deep down in DAG (note, these messages are linked to their children through immutable references, so we cannot avoid RefCel).

• The aggregators should be prunable and restorable. Meaning the containers should have function prune or some such that erases its content. If later this message is used again (e.g. a new message comes in that refers to it as parent) then we should be able to "restore" the content by recomputing it from the parents.

• Implement "pruner" a struct that will monitor the usage of the DAG and prune rarely used messages.

Note, we might not need to implement this for MVB, because TxFlow is getting restarted for each BeaconChain block.