ledgerwatch / erigon Goto Github PK

Ethereum implementation on the efficiency frontier

License: GNU Lesser General Public License v3.0

Dockerfile 0.05% Makefile 0.12% Go 96.26% Shell 0.16% Python 0.02% Solidity 0.45% Assembly 0.23% C 0.20% JavaScript 0.26% PowerShell 0.12% HTML 0.02% CSS 0.02% C++ 2.10%

ethereum blockchain

erigon's People

Contributors

Stargazers

Watchers

Forkers

tenfinney wl-chains kidmam carver matthalp-zz giulio2002 shiqinfeng1 ryanschneider thomasmodeneis timjp87 poemm s1na zhea gballet loganbek lucyzhang314 quilt rjdrost g-r-a-n-t satoshiandkin silentonecodes ucwong tjayrush hugo-dc pkrasam natebolam amxx askalexsharov dysnix renlord mrchico sc0vu lukaszrozmej hmijail ashwinrz lightclient crypt1d root-servers magicking canepat semenka 3esmit yuval-k guifel kimkyungup minxinping0105 sukhavati-labs gakonst fxfactorial pdaian contractshark qjawe ggarcia93 solaraww lundgren lithp transmissions11 lefterisjp alexkeating livethelifetv xeoskr sraka1 heipacker karlfloersch italoacasas roninkaizen advaithd kushalbabel baksten banteg ghettodev holiman trikunai mempirate bitbasebit coinhu tynes wmitsuda twilightrus vsvin covalenthq izayl zzyalbert suryatmodulus oeeckhoutte ricardolyn blockdaemon arielelp sambacha yonimo braman09 timdaub maticnetwork jupiterm floatingupstream the-coomer annadanchenko evrice riptl pleasecoding

erigon's Issues

Fix support for Ancient database in turbo-geth

There is a support for Ancient database in go-ethereum that requires some refactoring of the database code.

Misleading Log "Impossible reorg, please file an issue"

INFO [07-17|14:22:40.264] Database size=449657688064 written=22122497
INFO [07-17|14:22:45.825] Rewinding to block=8168731
Rewinding from block 8168739 to block 8168731
reorg 8168739 8168731
ERROR[07-17|14:22:46.017] Impossible reorg, please file an issue oldnum=8168731 oldhash=61753c…32cb12 newnum=8168731 newhash=61753c…32cb12
INFO [07-17|14:22:53.776] Imported new chain segment blocks=11 txs=1725 mgas=87.858 elapsed=8.131s mgasps=10.804 number=8168742 hash=2e878a…7e67f8 batch=14005 age=6m20s

Integrate blocking profiler

https://github.com/golang/go/wiki/Performance#blocking-profiler

Remove storage roots from history bucket

By history bucket I mean the one named hAT

Make encoding of values in the SUFFIX (changeset) bucket ordered by keys and binary-searchable

We are currently in the process of extending the values in the SUFFUX bucket to contain not just the list of keys that have changed in given block, but also previous values (before the change). This allows decoupling of rewinding from the history of state buckets hAT and hST, making history of state optional, and also simplifying the rewinding.

We should go further and make another two improvements in the way we encode the list of key-value pairs:

Make the list sorted by keys. This will allow more efficient merging of subsequent change-sets without deserialising every change-set first (in rewindData).
Prepend (or append) the list by the positional index, so that the binary search on the keys is possible. We expect the values become quite large (100ks - megabytes potentially). Binary search will be useful to implement historical resolver `trie.Resolver' in the way which is not coupled to the history of state buckets 'hAT' and 'hST'. The idea is to perform rewind of a sub-trie, and for that, we would need to be able to efficiently derive change-sets only for certain range of keys. Start of the range can be found by binary search and then keys can be iterated until we reach the end of the desired range.

Not to wrap values into RLP when storing the valueNode in the trie

Before we performed state trie unification and therefore created a separate leaf node type accountNode, simple value nodes and account nodes needed different amount of RLP "wrapping" (The quirk of yellow paper). Now that that their are separate, we should remove RLP wrapping from valueNode leaves. This will simplify code in some parts, and also reduce number of allocation due to wrapping/up-wrapping.
Issue #118 was a result of extra wrapping applied erroneously

Add badgerDb as an alternative to BoltDb

Badger DB satisfies the main requirements to a database for Turbo-Geth, namely:

ACID transactions. This is important for preventing database corruption due to panic, and abrupt shutdowns
MVCC (Multi-Version Concurrency Control). This is important for our project of creating a read-only node (with remote over-the-network access to the database of the write-node).

The main difference between BoltDB and BadgerDB is that BoltDB uses B+trees as the data structure, whereas BadgerDB - LSM (Log Structured Merge) trees. We have reasons to believe that LSM trees can perform better with the typical workload of syncing of Turbo-Geth node.

Godoc

We want automatic documentation as it present at https://godoc.org/github.com/ethereum/go-ethereum

[EIP 2031] Net transaction counter

Implements https://eips.ethereum.org/EIPS/eip-2031

Fix TestCopy datarace

==================
WARNING: DATA RACE
Write at 0x00c0001f3558 by goroutine 7:
  github.com/ledgerwatch/turbo-geth/trie.(*Trie).insert()
      /home/b00ris/go/src/github.com/ledgerwatch/turbo-geth/trie/trie.go:319 +0x1e08
  github.com/ledgerwatch/turbo-geth/trie.(*Trie).insert()
      /home/b00ris/go/src/github.com/ledgerwatch/turbo-geth/trie/trie.go:380 +0x929
  github.com/ledgerwatch/turbo-geth/trie.(*Trie).insert()
      /home/b00ris/go/src/github.com/ledgerwatch/turbo-geth/trie/trie.go:420 +0x1038
  github.com/ledgerwatch/turbo-geth/trie.(*Trie).insert()
      /home/b00ris/go/src/github.com/ledgerwatch/turbo-geth/trie/trie.go:420 +0x1038
  github.com/ledgerwatch/turbo-geth/trie.(*Trie).Update()
      /home/b00ris/go/src/github.com/ledgerwatch/turbo-geth/trie/trie.go:131 +0x19b
  github.com/ledgerwatch/turbo-geth/core/state.(*TrieDbState).computeTrieRoots()
      /home/b00ris/go/src/github.com/ledgerwatch/turbo-geth/core/state/database.go:551 +0xf3b
  github.com/ledgerwatch/turbo-geth/core/state.(*TrieDbState).ComputeTrieRoots()
      /home/b00ris/go/src/github.com/ledgerwatch/turbo-geth/core/state/database.go:296 +0x6f
  github.com/ledgerwatch/turbo-geth/core/state.TestCopy()
      /home/b00ris/go/src/github.com/ledgerwatch/turbo-geth/core/state/statedb_test.go:611 +0x181e
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:827 +0x162

Previous read at 0x00c0001f3558 by goroutine 16:
  github.com/ledgerwatch/turbo-geth/trie.(*Trie).NeedResolution()
      /home/b00ris/go/src/github.com/ledgerwatch/turbo-geth/trie/trie.go:167 +0x25e
  github.com/ledgerwatch/turbo-geth/core/state.(*TrieDbState).resolveAccountTouches()
      /home/b00ris/go/src/github.com/ledgerwatch/turbo-geth/core/state/database.go:438 +0x142
  github.com/ledgerwatch/turbo-geth/core/state.(*TrieDbState).computeTrieRoots()
      /home/b00ris/go/src/github.com/ledgerwatch/turbo-geth/core/state/database.go:490 +0x19d
  github.com/ledgerwatch/turbo-geth/core/state.(*TrieDbState).ComputeTrieRoots()
      /home/b00ris/go/src/github.com/ledgerwatch/turbo-geth/core/state/database.go:296 +0x6f
  github.com/ledgerwatch/turbo-geth/core/state.TestCopy.func1()
      /home/b00ris/go/src/github.com/ledgerwatch/turbo-geth/core/state/statedb_test.go:589 +0x1d5

Goroutine 7 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:878 +0x650
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:1119 +0xa8
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:827 +0x162
  testing.runTests()
      /usr/local/go/src/testing/testing.go:1117 +0x4ee
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:1034 +0x2ee
  main.main()
      _testmain.go:68 +0x221

Goroutine 16 (running) created at:
  github.com/ledgerwatch/turbo-geth/core/state.TestCopy()
      /home/b00ris/go/src/github.com/ledgerwatch/turbo-geth/core/state/statedb_test.go:582 +0x1688
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:827 +0x162

Better interface to GenerateChain in chain_makers.go and fix tests

Some of the tests in core/blockchain_test.go are skipped, but can be enabled if there was a better way of specifying a snapshot of the database from which to build the chain

Make Turbogeth imports compatible with go-ethereum

When rebasing to a new version of geth, a lot of conflicts happen because of different import names.

When a new import added upstream, git gets confused. Resolving these merges is trivial, but annoying and can lead to occasionally importing actual unpatched go-ethereum module.

Crash during sync in the trie cleaning

Backtrace

INFO [10-04|12:27:59.317] Memory                                   nodes=4999974 alloc=14091714 sys=25457625 numGC=287
INFO [10-04|12:27:59.317] Database                                 size=142020145152 written=570843137
INFO [10-04|12:27:59.358] Imported new chain segment               blocks=31  txs=3409  mgas=173.164  elapsed=11.556s   mgasps=14.984  number=5104347 hash=c38607…61a26
7 batch=584    age=1y7mo3w
INFO [10-04|12:28:07.370] Imported new chain segment               blocks=130 txs=14839 mgas=804.555  elapsed=8.011s    mgasps=100.422 number=5104477 hash=224281…77f2a
1 batch=84155  age=1y7mo3w
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x8c090e]
goroutine 53760 [running]:
github.com/ledgerwatch/turbo-geth/trie.(*hasher).hashInternal(0xc2dbe44000, 0x0, 0x0, 0xc0f6d65e00, 0xc07a77fd60, 0x20, 0x20, 0x0, 0x45e491)
        /home/akhounov/workspace/src/github.com/ledgerwatch/turbo-geth/trie/hasher.go:80 +0x8e
github.com/ledgerwatch/turbo-geth/trie.(*hasher).hash(...)
        /home/akhounov/workspace/src/github.com/ledgerwatch/turbo-geth/trie/hasher.go:69
github.com/ledgerwatch/turbo-geth/trie.(*Trie).unload(0xc000ec5480, 0xc28b06e400, 0x40, 0x40, 0xc2dbe44000)
        /home/akhounov/workspace/src/github.com/ledgerwatch/turbo-geth/trie/trie.go:1255 +0x4bd
github.com/ledgerwatch/turbo-geth/trie.pruneMap(0xc000ec5480, 0xc227afef70, 0xc2dbe44000, 0x41)
        /home/akhounov/workspace/src/github.com/ledgerwatch/turbo-geth/trie/trie_pruning.go:160 +0x232
github.com/ledgerwatch/turbo-geth/trie.(*TriePruning).PruneToTimestamp(0xc000554c60, 0xc000ec5480, 0x4d1fcc)
        /home/akhounov/workspace/src/github.com/ledgerwatch/turbo-geth/trie/trie_pruning.go:189 +0x266
github.com/ledgerwatch/turbo-geth/trie.(*TriePruning).PruneTo(0xc000554c60, 0xc000ec5480, 0x4c4b40, 0x1)
        /home/akhounov/workspace/src/github.com/ledgerwatch/turbo-geth/trie/trie_pruning.go:216 +0xbf
github.com/ledgerwatch/turbo-geth/core/state.(*TrieDbState).PruneTries(0xc0000e2a10, 0x216e7000)
        /home/akhounov/workspace/src/github.com/ledgerwatch/turbo-geth/core/state/database.go:1043 +0x80
github.com/ledgerwatch/turbo-geth/core.(*BlockChain).insertChain(0xc000170400, 0x1303da0, 0xc187943740, 0xc540418000, 0xb79, 0xb79, 0x1, 0x0, 0x0, 0x0, ...)
        /home/akhounov/workspace/src/github.com/ledgerwatch/turbo-geth/core/blockchain.go:1360 +0x25c0
github.com/ledgerwatch/turbo-geth/core.(*BlockChain).InsertChain(0xc000170400, 0xc540418000, 0xb79, 0xb79, 0x2065648, 0x0, 0x0)
        /home/akhounov/workspace/src/github.com/ledgerwatch/turbo-geth/core/blockchain.go:1068 +0xe04
github.com/ledgerwatch/turbo-geth/eth/downloader.(*Downloader).importBlockResults(0xc00025a6e0, 0xc53ff4e000, 0xb79, 0xb79, 0xb79, 0x0)
        /home/akhounov/workspace/src/github.com/ledgerwatch/turbo-geth/eth/downloader/downloader.go:1471 +0x538
github.com/ledgerwatch/turbo-geth/eth/downloader.(*Downloader).processFullSyncContent(0xc00025a6e0, 0xc0001086c0, 0xc0ecee5798)
        /home/akhounov/workspace/src/github.com/ledgerwatch/turbo-geth/eth/downloader/downloader.go:1445 +0x79
github.com/ledgerwatch/turbo-geth/eth/downloader.(*Downloader).spawnSync.func1(0xc00025a6e0, 0xc244e00d20, 0xc43c514a10)
        /home/akhounov/workspace/src/github.com/ledgerwatch/turbo-geth/eth/downloader/downloader.go:478 +0x59
created by github.com/ledgerwatch/turbo-geth/eth/downloader.(*Downloader).spawnSync
        /home/akhounov/workspace/src/github.com/ledgerwatch/turbo-geth/eth/downloader/downloader.go:478 +0xaf
(END)

letter-coded sync modes instead of full, archive etc.

Some database buckets are mandatory for successful sync and block processing. For example, AT (current state of accounts), ST (current state of contract storage), SUFFIX (changesets used for rewinding). Other buckets are optional and only add value for certain users. We would like users to have flexibility to pick and choose those buckets in any combination. Users may specify the mode of operation as a combination of one-letter flags (similar to unix tar command). Here are suggested flags:

r Transaction receipts. Currently turned off by default. If turned on, receipts are written to a special bucket, which is then used to server getTransactionReceipt and getLogs RPC commands. In blockchain.go this is set by EnableReceipts(true)
h History of state. These are buckets hAT (history of state of accounts) and hST (history of state of contract storage). Although these buckets contain the same information as the SUFFIX (change-set) bucket, it is organised for quick access using addressHash | blockNr composite key for accounts, and addressHash | incarnation | blockNr composite key for contract storage. Bucket SUFFIX is organised for quick access for composite keys "hAT" | blockNr and "hST" | blockNr. History of state is used for answering various RPC queries related to account balances, contract storage contents at some point in the past.
t Index allowing to look up block number and transaction index given a transaction hash. It is used by RPC requests such as getTransaction, because it allows retrieving the body of the transaction from the bucket with blocks
p preimages for address hashes and storage key hashes. This information is needed by RPC queries such as getModifiedAccounts and getStorageRangeAt.
Block headers and bodies? In order to perform reorgs, some short history of alternative blocks and headers is required.

Some examples of which modes might be useful for different use-cases:

Miners would prefer to turn all the flags off, they only care about the current state and occasional reorgs. Their priority - low latency of block processing, and lowering I/O would definitely help
Exchanges would need preimages (p) and history (h) to be able to watch for deposits. If they use logs (events) to figure out token transfers, they might also want to turn on receipts (r) for better performance
Block explorers might want to turn everything on
Dapp Web sites would want to turn on receipts (r) for better scanning of the logs (lots of dapp use logs as a substitute for storage).

Unstable linters in CI

Problems

lint step in CircleCI is unstable
CircleCI kills linters job because linters consumes more 4gb of memory. (https://circleci.com/gh/ledgerwatch/turbo-geth/251)

Possible solutions

migrate to TravisCI
add more steps for lint
More memory for circleCI account
Disable some linters

Not Connecting to Parity Peers

DEBUG[07-04|05:45:50.848] Adding p2p peer name=Parity-Ethereum/v2.5... addr=67.110.215.92:5
6852 peers=15
DEBUG[07-04|05:45:50.848] Ethereum peer connected id=904a58a02a6ce296 conn=inbound name=Parity-Ethereum/v2.5.1-beta-adabd81-20190514/x86_64-linux-gnu/rustc1.34.1
DEBUG[07-04|05:45:50.848] Failed to read Status msg id=904a58a02a6ce296 conn=inbound err=EOF
DEBUG[07-04|05:45:50.848] Ethereum handshake failed id=904a58a02a6ce296 conn=inbound err=EOF
DEBUG[07-04|05:45:50.848] Removing p2p peer id=904a58a02a6ce296 conn=inbound duration=325.273µs peers=14 req=false err=EOF
DEBUG[07-04|05:45:50.849] Failed to send Status msg id=904a58a02a6ce296 conn=inbound err="write tcp 10.128.0.7:30303->67.110.215.92:56852: use of closed network connection"

(built from red-queen @ commit 1b2c836 )

Reorganise the way contract codes are stored in the database, and then remove codeHash from hAT bucket

Presently, the contract codes are stored in the bucket CODE, and it is a simple mapping of code hashes to bytecodes. We would like to change that to a mapping of a composite key contractAddrHash | incarnation to the bytecode. That would allow us to remove codeHash component from the hAT bucket (which is the history of state of accounts), therefore reducing the size of the bucket and write I/O to maintain it. It will also allow extension of pruning to contract codes.

The downside might be potential code duplication. If such downside to be avoided, we can use the indirect mapping contractAddrHash | incarcation to the code hash, and then add some kind of reference counter to the existing mapping of codeHash to bytecode (to be able to remove it when all the contracts using such bytecode have been pruned).

[EIP 2026] Fixed rent prepayments

implements https://eips.ethereum.org/EIPS/eip-2026

Merge files trie/structural.go and trie/structural_2.go

[EIP 2027] Basic unit tests

Fix stCreate2/create2collisionStorage tests

st.fails(`^stCreate2/create2collisionStorage.json/Constantinople/0`, "work in progress")
st.fails(`^stCreate2/create2collisionStorage.json/Constantinople/1`, "work in progress")
st.fails(`^stCreate2/create2collisionStorage.json/Constantinople/1`, "work in progress")
st.fails(`^stCreate2/create2collisionStorage.json/Constantinople/2`, "work in progress")
st.fails(`^stCreate2/create2collisionStorage.json/ConstantinopleFix/0`, "work in progress")
st.fails(`^stCreate2/create2collisionStorage.json/ConstantinopleFix/1`, "work in progress")
st.fails(`^stCreate2/create2collisionStorage.json/ConstantinopleFix/2`, "work in progress")

from https://github.com/ledgerwatch/turbo-geth/blob/master/tests/state_test.go

Looking ways to collaborate with you

Hi there,

I want to collaborate with this project and I'm looking for some starting issues or ways to collaborate. I'm a programmer at Decentraland and we are interested in helping the development of storage rent and other eth 1x improvements.

I read about the state rent proposal and I understand the major points

Last week I could start syncing the node, but it would fail to validate all blocks on restart. Today it doesn't event start to download blocks. Neither in master nor in the eip-2027 branch. Probably there are different ways for testing the node that do not require to syncing it and you may help me pointing those?

Or maybe you know better how I can help you guys

Thanks,

Fix concurrent map writes error


goroutine 48 [running]:
runtime.throw(0x72f8b2, 0x15)
	/usr/local/go/src/runtime/panic.go:617 +0x72 fp=0xc00011a758 sp=0xc00011a728 pc=0x42f862
runtime.mapassign_fast64(0x6cffe0, 0xc000013050, 0x0, 0x5dfd9e)
	/usr/local/go/src/runtime/map_fast64.go:101 +0x35f fp=0xc00011a798 sp=0xc00011a758 pc=0x4135cf
github.com/ledgerwatch/turbo-geth/core/state.(*TrieDbState).leftGeneration(...)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/core/state/database.go:686
github.com/ledgerwatch/turbo-geth/core/state.(*TrieDbState).leftGeneration-fm(0x0)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/core/state/database.go:684 +0x49 fp=0xc00011a7c8 sp=0xc00011a798 pc=0x6747d9
github.com/ledgerwatch/turbo-geth/trie.(*duoNode).updateT(0xc0005269c0, 0x1, 0xc0005120d0, 0xc0005120e0)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/trie/node.go:334 +0x48 fp=0xc00011a7e0 sp=0xc00011a7c8 pc=0x5dfc68
github.com/ledgerwatch/turbo-geth/trie.(*Trie).insert(0xc000108100, 0x7a8d00, 0xc0005269c0, 0xc00082acd0, 0x41, 0x41, 0x2, 0x7a8f40, 0xc000820c20, 0xc00011ae90, ...)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/trie/trie.go:1227 +0xa43 fp=0xc00011a9e8 sp=0xc00011a7e0 pc=0x5fb103
github.com/ledgerwatch/turbo-geth/trie.(*Trie).insert(0xc000108100, 0x7a8d60, 0xc0005d6c60, 0xc00082acd0, 0x41, 0x41, 0x1, 0x7a8f40, 0xc000820c20, 0xc00011ae90, ...)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/trie/trie.go:1329 +0x633 fp=0xc00011abf0 sp=0xc00011a9e8 pc=0x5facf3
github.com/ledgerwatch/turbo-geth/trie.(*Trie).insert(0xc000108100, 0x7a8d60, 0xc000511760, 0xc00082acd0, 0x41, 0x41, 0x0, 0x7a8f40, 0xc000820c20, 0xc00011ae90, ...)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/trie/trie.go:1329 +0x633 fp=0xc00011adf8 sp=0xc00011abf0 pc=0x5facf3
github.com/ledgerwatch/turbo-geth/trie.(*Trie).Update(0xc000108100, 0xc00011b200, 0x20, 0x20, 0xc00082ac80, 0x48, 0x48, 0x1)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/trie/trie.go:808 +0x178 fp=0xc00011af30 sp=0xc00011adf8 pc=0x5f7c28
github.com/ledgerwatch/turbo-geth/core/state.(*TrieDbState).computeTrieRoots(0xc000718c60, 0xc000501901, 0x0, 0x3e, 0x8060104, 0x3e, 0xffffffffffffffff)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/core/state/database.go:498 +0x1a57 fp=0xc000131c10 sp=0xc00011af30 pc=0x656977
github.com/ledgerwatch/turbo-geth/core/state.(*TrieDbState).ComputeTrieRoots(0xc000718c60, 0xc000010201, 0x7a7e80, 0xc000010200, 0x0, 0x0)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/core/state/database.go:286 +0x34 fp=0xc000131c58 sp=0xc000131c10 pc=0x654b84
github.com/ledgerwatch/turbo-geth/core/state.TestCopy(0xc00010a300)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/core/state/statedb_test.go:190 +0x98f fp=0xc000131fa8 sp=0xc000131c58 pc=0x66c71f
testing.tRunner(0xc00010a300, 0x73aec0)
	/usr/local/go/src/testing/testing.go:865 +0xc0 fp=0xc000131fd0 sp=0xc000131fa8 pc=0x4e2250
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000131fd8 sp=0xc000131fd0 pc=0x45da11
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:916 +0x35a

goroutine 1 [chan receive]:
testing.(*T).Run(0xc00010a300, 0x72bbd6, 0x8, 0x73aec0, 0x479101)
	/usr/local/go/src/testing/testing.go:917 +0x381
testing.runTests.func1(0xc00010a200)
	/usr/local/go/src/testing/testing.go:1157 +0x78
testing.tRunner(0xc00010a200, 0xc0000e1e30)
	/usr/local/go/src/testing/testing.go:865 +0xc0
testing.runTests(0xc00000f120, 0xbad040, 0xc, 0xc, 0x0)
	/usr/local/go/src/testing/testing.go:1155 +0x2a9
testing.(*M).Run(0xc000108180, 0x0)
	/usr/local/go/src/testing/testing.go:1072 +0x162
main.main()
	_testmain.go:64 +0x13e

goroutine 49 [runnable]:
github.com/ledgerwatch/turbo-geth/rlp.(*encbuf).encodeString(0xc00082c000, 0xc0004a867e, 0x2, 0x2)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/rlp/encode.go:201 +0x1ba
github.com/ledgerwatch/turbo-geth/rlp.writeBigInt(0xc00038c7c0, 0xc00082c000, 0x196, 0x40e801)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/rlp/encode.go:435 +0x199
github.com/ledgerwatch/turbo-geth/rlp.writeBigIntPtr(0x728320, 0xc000297e20, 0x196, 0xc00082c000, 0x728320, 0xc000297e20)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/rlp/encode.go:421 +0x10d
github.com/ledgerwatch/turbo-geth/rlp.makeStructWriter.func1(0x6fc7c0, 0xc000297e18, 0x199, 0xc00082c000, 0xc000297e18, 0x199)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/rlp/encode.go:535 +0x149
github.com/ledgerwatch/turbo-geth/rlp.makePtrWriter.func4(0x6ab120, 0xc000297e18, 0x16, 0xc00082c000, 0x0, 0x0)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/rlp/encode.go:578 +0xb1
github.com/ledgerwatch/turbo-geth/rlp.(*encbuf).encode(0xc00082c000, 0x6ab120, 0xc000297e18, 0x706b00, 0xc00082c000)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/rlp/encode.go:187 +0x135
github.com/ledgerwatch/turbo-geth/rlp.EncodeToBytes(0x6ab120, 0xc000297e18, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/rlp/encode.go:101 +0xe4
github.com/ledgerwatch/turbo-geth/core/state.(*TrieDbState).computeTrieRoots(0xc000184000, 0xc0004f2001, 0x0, 0x13, 0x7060107, 0x13, 0xffffffffffffffff)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/core/state/database.go:494 +0x19ee
github.com/ledgerwatch/turbo-geth/core/state.(*TrieDbState).ComputeTrieRoots(0xc000184000, 0xc000398001, 0x7a7e80, 0xc000398000, 0x0, 0x0)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/core/state/database.go:286 +0x34
github.com/ledgerwatch/turbo-geth/core/state.TestCopy.func1(0xc000506000, 0xc000184000, 0xc0003f6120)
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/core/state/statedb_test.go:184 +0x7e
created by github.com/ledgerwatch/turbo-geth/core/state.TestCopy
	/go/src/github.com/ledgerwatch/turbo-geth/build/_workspace/src/github.com/ledgerwatch/turbo-geth/core/state/statedb_test.go:182 +0x929

[EIP 2029] State counters contract

We need to implement https://eips.ethereum.org/EIPS/eip-2029

Remove vendor and fix tests for go 1.12.9

Benchmark Accounts package

Pruning proceeds ahead of last sync blocks

When syncing to Ropsten, where it takes a long time for sync to start, I observed that pruning has started ahead of sync:

INFO [10-16|10:46:00.988] Initialising Ethereum protocol           versions="[63 62]" network=3
INFO [10-16|10:46:01.010] Loaded most recent local header          number=0 hash=419410…ca4a2d td=1048576 age=50y6mo5d
INFO [10-16|10:46:01.011] Loaded most recent local full block      number=0 hash=419410…ca4a2d td=1048576 age=50y6mo5d
INFO [10-16|10:46:01.011] Loaded most recent local fast block      number=0 hash=419410…ca4a2d td=1048576 age=50y6mo5d
INFO [10-16|10:46:01.011] Pruner started 
INFO [10-16|10:46:01.011] Loaded local transaction journal         transactions=0 dropped=0
INFO [10-16|10:46:01.012] Regenerated local transaction journal    transactions=0 accounts=0
INFO [10-16|10:46:01.012] Initialising Firehose protocol           versions=[1]
INFO [10-16|10:46:01.088] New local node record                    seq=23 id=a3c8dc9a49dedf3b ip=127.0.0.1 udp=30303 tcp=30303
INFO [10-16|10:46:01.088] Started P2P networking                   self=enode://d3228a32673a4c6ec8556a6796bf3a4ea94d288aed9db43a98d7ba742abd296624b2c7f02bd3ad88250ce4b74a777dfdb30fef0a4cad7f484b8c748221f6c626@127.0.0.1:30303
INFO [10-16|10:46:01.091] IPC endpoint opened                      url=/Users/alexeyakhunov/Library/Ethereum/testnet/geth.ipc
INFO [10-16|10:51:01.036] Save last pruned block num               num=3000
INFO [10-16|10:56:01.028] Save last pruned block num               num=6000
INFO [10-16|11:01:01.016] Save last pruned block num               num=9000
INFO [10-16|11:06:01.007] Save last pruned block num               num=12000

The reason is underflow when computing diff in this code:

func calculateNumOfPrunedBlocks(curentBlock, lastPrunedBlock uint64, blocksBeforePruning uint64, blocksBatch uint64) (uint64, uint64, bool) {
	diff := curentBlock - lastPrunedBlock - blocksBeforePruning
	switch {
	case diff >= blocksBatch:
		return lastPrunedBlock, lastPrunedBlock + blocksBatch, true
	case diff > 0 && diff < blocksBatch:
		return lastPrunedBlock, lastPrunedBlock + diff, true
	default:
		return lastPrunedBlock, lastPrunedBlock, false
	}
}

Also there is a typo in the word curentBlock :)

Incarnations not passed in certain places

At least I can see missing logic for incarnation in the following files:

core/state/readonly.go
eth/handler.go

Visualisation of database tables after each change of state and highlighting of the changes

Now that we can visualise the database buckets, we would like to do this after each change of state in the initialState1 function, and highlight what changed every time. Highlighting is achieved by using highlighted parameter of the Horizontal function.
In order to highlight the differences, you would probably want to snapshot the database (using MemCopy), and then write a function that compares the current state with the snapshot.

Switch historical trie.Resolver from using hAT & hST bucket to using SUFFIX (changeset) bucket

Currently trie.Resolver supports historical mode, where it reads the range of keys from the history "As Of" given block number. The efficiently of such operation declines with the growth of the historical buckets.

Alternative approach, which should work more efficiently for the cases where the "As Of" point is not too far into the past, is to utilise change-sets in the SUFFIX bucket, and effectively rewind the range of the current state while resolving. This should help answer queries related to snapshot sync Red-Queen/Firehose protocol, as well as light server support in the future.

Issue #123 provides a way to perform such partial rewind efficiently because binary search on the change sets becomes possible

Fix RevertInCreateInInit test

st.fails(`^stRevertTest/RevertInCreateInInit.json/Byzantium/0`, "work in progress")
st.fails(`^stRevertTest/RevertInCreateInInit.json/Constantinople/0`, "work in progress")
st.fails(`^stRevertTest/RevertInCreateInInit.json/ConstantinopleFix/0`, "work in progress")

from https://github.com/ledgerwatch/turbo-geth/blob/master/tests/state_test.go

Sporadic failure in TestFakedSyncProgress62

https://circleci.com/gh/ledgerwatch/turbo-geth/17

--- FAIL: TestFakedSyncProgress62 (1.73s)
downloader_test.go:1416: final progress mismatch:
have {StartingBlock:0 CurrentBlock:1004 HighestBlock:1003 PulledStates:0 KnownStates:0}
want {StartingBlock:0 CurrentBlock:1003 HighestBlock:1003 PulledStates:0 KnownStates:0}

Fix InitCollision ConstantinopleFix tests

st.fails(^stSStoreTest/InitCollision.json/ConstantinopleFix/0, "work in progress")
st.fails(^stSStoreTest/InitCollision.json/ConstantinopleFix/1, "work in progress")
st.fails(^stSStoreTest/InitCollision.json/ConstantinopleFix/3, "work in progress")
from https://github.com/ledgerwatch/turbo-geth/blob/master/tests/state_test.go

MemCopy data race

Restore full sync mode

Restore db pruning and full sync mode

Replace red-black trees with simple maps in ethdb.mutation

Originally, red-black trees were put there to support iteration (Walk function) over the mutation. However, this operation has been removed, and iterations are now only supported over the committed databases (and not mutation). There is no good reason to keep using red black trees instead of simple maps anymore

Check and fix reorg

for full sync archive node
for full sync -noHistory

archive mode

INFO [09-18|15:46:37.141] Upgrading chain index                    type=bloombits percentage=96
INFO [09-18|15:46:45.351] Imported new chain segment               blocks=3002 txs=1525  mgas=247.621  elapsed=8.001s    mgasps=30.946  number=223069 hash=7aba52…bd44e0 batch=28692  age=2y9mo6d
INFO [09-18|15:46:47.851] Finished upgrading chain index           type=bloombits
INFO [09-18|15:47:01.878] Memory                                   nodes=188300 alloc=725282  sys=1321001 numGC=630
INFO [09-18|15:47:01.878] Database                                 size=1209487360 written=107163649
INFO [09-18|15:47:01.913] Imported new chain segment               blocks=3099 txs=5324  mgas=783.476  elapsed=16.562s   mgasps=47.305  number=226168 hash=4a8b75…af1fe3 batch=276    age=2y9mo6d
INFO [09-18|15:47:05.460] Imported new chain segment               blocks=2091 txs=4035  mgas=596.935  elapsed=3.546s    mgasps=168.307 number=228259 hash=2795d9…b7c1d8 batch=59592  age=2y9mo5d
INFO [09-18|15:47:19.710] Memory                                   nodes=192546 alloc=701734  sys=1456301 numGC=632
INFO [09-18|15:47:19.710] Database                                 size=1209487360 written=72851457
INFO [09-18|15:47:19.742] Upgrading chain index                    type=bloombits percentage=96
INFO [09-18|15:47:28.189] Imported new chain segment               blocks=1827 txs=3326  mgas=409.059  elapsed=8.004s    mgasps=51.101  number=230086 hash=0f8b88…09fed1 batch=35410  age=2y9mo5d
INFO [09-18|15:47:28.209] Finished upgrading chain index           type=bloombits
INFO [09-18|15:47:40.801] Memory                                   nodes=201048 alloc=697643  sys=1456301 numGC=639
INFO [09-18|15:47:40.801] Database                                 size=1249263616 written=104996865
INFO [09-18|15:47:40.802] Imported new chain segment               blocks=676  txs=3503  mgas=661.853  elapsed=12.612s   mgasps=52.475  number=230762 hash=ec974c…e25486 batch=11     age=2y9mo5d
INFO [09-18|15:47:40.955] Rewinding                                to block=228259
ERROR[09-18|15:47:44.342] Incorrect rewinding                      root=833633f1876d00cba03887e9b7bbbd11cfb583d7d0e0b70adbefef9921c263b1 expected=fe568ba6a8fb7b300774993e60465ce8ced71a583fc0b68faf56444544cebc9c
WARN [09-18|15:47:44.345] Synchronisation failed, dropping peer    peer=774a4679069c07a4 err="retrieved hash chain is invalid"
INFO [09-18|15:47:47.708] Creating IntraBlockState from latest state block=230881
WARN [09-18|15:47:47.810] Synchronisation failed, dropping peer    peer=92570a250f771747 err="retrieved hash chain is invalid"
INFO [09-18|15:53:50.693] Rewinding                                to block=230881
ERROR[09-18|15:53:50.719] Incorrect rewinding                      root=d7f66d90a145bcdee58ca1d36f2be53e731ef303aea8bc7ee8358725db857dd5 expected=fe568ba6a8fb7b300774993e60465ce8ced71a583fc0b68faf56444544cebc9c
WARN [09-18|15:53:50.719] Synchronisation failed, dropping peer    peer=190b5d40eb03cd59 err="retrieved hash chain is invalid"

and then it stuck.

Fix InitCollision tests

st.fails(`^stSStoreTest/InitCollision.json/Constantinople/0`, "work in progress")
st.fails(`^stSStoreTest/InitCollision.json/Constantinople/1`, "work in progress")
st.fails(`^stSStoreTest/InitCollision.json/Constantinople/2`, "work in progress")
st.fails(`^stSStoreTest/InitCollision.json/Constantinople/3`, "work in progress")

from https://github.com/ledgerwatch/turbo-geth/blob/master/tests/state_test.go

Fix ExtCodeHash/dynamicAccountOverwriteEmpty tests

st.fails(`^stExtCodeHash/dynamicAccountOverwriteEmpty.json/Constantinople/0`, "work in progress")
st.fails(`^stExtCodeHash/dynamicAccountOverwriteEmpty.json/ConstantinopleFix/0`, "work in progress")

from https://github.com/ledgerwatch/turbo-geth/blob/master/tests/state_test.go

Remove Copy member functions from IntraBlockState

Since there is no efficient way of copying intra-block-state (for the purpose of concurrent access, for example), it makes sense to remove Copy functions rather than leaving them incorrect, but present.
This will also illuminate the need to fix the miner code

[EIP 2035] Repricing SLOAD and SSTORE to pay for block proofs

implements https://eips.ethereum.org/EIPS/eip-2035

Investigate a broken snapshot after archive sync

After having performed archive sync, I transmitted the database (around 500Gb) to another computer and tried to resume sync from it.
However, the sync very soon failed because the resolution of one of the contracts returned a wrong storageRoot.
The success of transmission has been checked by performing SHA256 on both files, and the results matched.
The plan is to compare storageRoot for all contracts with the computed values, then choose the smallest contract where the mismatch happens, and attempt to find the cause for the mismatch.

Fix mining code

Mining code is currently disabled, because mining logic needs special handling. Since the state in Turbo-Geth is singleton, miner has to share it with the block processing, rather than relying on the thread-safe copy. The solution should be similar to #111, essentially computing the state root of the mined block without modifying the state trie.

[EIP 2027] Net contract size accounting

Not to mutate state trie when verifying the block

When we import a block, we execute txs in it and mutate the trie. However, if the state root hash does not match, we need to undo the changes. This does not happen now. Alternatively, we need to find a way to compute the state root without mutating the trie. This would also be important for mining.

Add visualisations of database tables to the visualisation of state tries

Please use the branch block_witness for reference until this PR is merged: #112

The file https://github.com/ledgerwatch/turbo-geth/blob/block_witness/cmd/pics/state.go contains the code that generates a series of illustrations used in this presentation (still needs to be turned to a documentation): https://github.com/AlexeyAkhunov/papers/blob/master/STARKs_for_Stateless_client_The_beginning.pdf

In order to run them, you can execute

make pics
./build/bin/pics -pic initial_state_1

You would need graphviz to be installed for this to work. If it worked, you will see lots of PNG files in the current directory.

After generating each picture, it would be great if we committed the changes to the database (db.Commit()), and then visualise the buckets in the database. The code that iterates over the buckets can be taken from here: https://github.com/ledgerwatch/turbo-geth/blob/block_witness/ethdb/memory_database.go (function MemCopy), but instead of inserting all the keys and values into a new (copy) database, we would try to visualise them, perhaps using the same primitives (coloured squares) as we use for the trie. Specifically, I would start with visualising the buckets AT (accounts) and ST. In order to see the correspondence between the entries in the database and the content of the state trie, the keys taken from the database need to be converted to HEX or QUAD encoding before being displayed, and perhaps also split into parts (for ST bucket).

Eventually, these new visualisations would become part of the programmer's guide: https://github.com/ledgerwatch/turbo-geth/blob/master/docs/programmers_guide/guide.md

split install command

Linters in CI

Eugene:

кстати, можно от таких штук прикрутить линтеров в CI. А чтоб не бороться со старым кодом, то линтеры можно запускать только для diff с текущим мастеров.

golangci, я думаю.
в make что-то типа
LATEST_COMMIT ?= $(shell git log -n 1 origin/master --pretty=format:"%H")
ifeq ($(LATEST_COMMIT),)
LATEST_COMMIT := $(shell git log -n 1 HEAD~1 --pretty=format:"%H")
endif

lint:
@echo "--> Running linter for code diff versus commit $(LATEST_COMMIT)"
@./build/bin/golangci-lint run --new-from-rev=$(LATEST_COMMIT)

lint-deps:
curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s -- -b ./build/bin v1.16.0
;

и .golangci.yml:

run:
deadline: 10m

linters:
enable-all: true
disable:
- lll
- dupl
- gochecknoglobals
- gocritic
- maligned
- goimports

linters-settings:
govet:
check-shadowing: true
golint:
min-confidence: 0
maligned:
suggest-new: true
goconst:
min-len: 2
min-occurrences: 2
misspell:
locale: US
gocyclo:
min-complexity: 35
gofmt:
auto-fix: false

месяц назад в форк для работы с geth добавлял :)

Fix Create2/RevertInCreateInInitCreate2 tests

st.fails(`^stCreate2/RevertInCreateInInitCreate2.json/Constantinople/0`, "work in progress")
st.fails(`^stCreate2/RevertInCreateInInitCreate2.json/ConstantinopleFix/0`, "work in progress")

from https://github.com/ledgerwatch/turbo-geth/blob/master/tests/state_test.go

Unstable TestWatchNoDir

? github.com/ledgerwatch/turbo-geth/accounts/external [no test files]
--- FAIL: TestWatchNoDir (12.70s)
account_cache_test.go:132:
got []
want [{[126 245 166 19 95 31 214 160 37 147 238 220 134 156 109 65 217 52 174 248] keystore:///tmp/eth-keystore-watch-test-8097-2334823778697134485/aaa}]
FAIL
FAIL github.com/ledgerwatch/turbo-geth/accounts/keystore 20.501s

https://circleci.com/gh/ledgerwatch/turbo-geth/401

ledgerwatch / erigon Goto Github PK

erigon's People

Contributors

Stargazers

Watchers

Forkers

erigon's Issues

Backtrace

Problems

Possible solutions

Recommend Projects

Recommend Topics

Recommend Org