EVM-based Blockchain Indexer, with historical data query & real-time notification support 😎
Home Page: https://itzmeanjan.github.io/ette/
License: Creative Commons Zero v1.0 Universal
Thanks for this piece!
I'm running the code in real-time mode and I'm not able to get the blocks that are being created. Continuously seeing error:
Failed to fetch block 14629780 : not found
Followed by:
Failed to fetch tx receipt [ block : 14629780 ] : not found
The block number and hash is valid and I can see in my Alchemy provider results are returned in raw format.
Any ideas?
how do change listners/indexer network
i want to index multiple chains so need to run multiple instance of ette on various network,
After signing the message with MetaMask, it does not go to dashboard, it only stays at the login screen.
How can i get past it. What might be wrong? Can i manually create APIkey, or disable the APIkey/metamask requirement?
Problem: when an unexpected block is received, ette exists with a log.Fatalf() call. That happens in two cases:
The latter states that ette bails because it basically stops relying on the node and that this is not a state of the art solution.
Suggestion: Change this behavior so that ette notifies the user of bad blocks and ignores them, instead of letting ette itself making the decision of stop relying on the node. On production, or pseudo-production environments, ette would be running on a supervised way, so that a supervisor would bring it up again in those situations. This means the strategy of firing a log.Fatalf() doesn't even have the expected behavior, but instead have only a bad side effect: it makes ette offline for a few seconds and then it gets back online, feeding from the same node again.
Tried ette for indexing the polygon blockchain but ette is getting killed around block 79000 or 89000 due to high memory usage. This always happens to me a every time I clear the db and run a fresh start of ette.
Instance Specs:
ubuntu 22.04
4 cpu / 8 gb ram
ette config:
RPCUrl=http://172.x.x.x:8545
WebsocketUrl=wss://xxxxx
PORT=1337
DB_USER=admin
DB_PASSWORD=redacted
DB_HOST=localhost
DB_PORT=5432
DB_NAME=polygon
RedisConnection=tcp
RedisAddress=127.0.0.1:6379
RedisPassword=redacted
Domain=localhost
Production=yes
EtteMode=1
EtteGraphQLPlayGround=yes
ConcurrencyFactor=2
BlockConfirmations=100
BlockRange=100
TimeRange=21600
SnapshotFile=snapshot.bin
Instance free memory before ette hangs up and getting killed, on htop ette occupies 85%+ of memory
Ette getting killed at block 89431
Returned Json is missing value and data fields./
{
"transactions": [
{
"hash": "0x133332d7a9ddbb587e091b5138087e9dfa88c30c00301c1d54606453c6f5ac11",
"from": "0x5073c1C5172cF29873802BC0cd8221c5Cef544E7",
"to": "0xD1d2BD3a92B063dFE2f6ba2ba8d67BBb3f4D96dd",
"value": "",
"data": "",
"gas": 21000,
"gasPrice": "10000000000",
"cost": "10000000000000000000",
"nonce": 0,
"state": 1,
"blockHash": "0x20d88bdcb99b87544d25bdd3a6618c015b1d66de59989b1b6996f34991409424"
}
]
}
Looking at the method here:
https://github.com/itzmeanjan/ette/blob/main/app/db/query.go#L365-L377
, we see it's not added in the DB query.
Is this a bug or is it intended?
hi,
Really impressive product and is exactly what I am looking for. Is there a way to push real time transcation notifications to kafka?
Hello, I'm having this problem (Failed to fetch block XX : not found) in ALL the blocks when I put the configuration: EtteMode=2 .
Blocks can never be fetch by the program.
I thought it was a concurrency issue but I've tried a local Ganache Blockchain and the exact same thing happens.
I leave you my .env file:
RPCUrl=http://23.88.42.126:8545
WebsocketUrl=ws://23.88.42.126:8545
#RPCUrl=https://mainnet.infura.io/v3/025d7ebf96134012b954d0491f384c53
#WebsocketUrl=wss://mainnet.infura.io/ws/v3/025d7ebf96134012b954d0491f384c53
PORT=7000
DB_USER=postgres
DB_PASSWORD=mypassGUS
DB_HOST=127.0.0.1
DB_PORT=5432
DB_NAME=ette
RedisConnection=tcp
RedisAddress=127.0.0.1:6379
RedisPassword=mypassGUS
Domain=localhost
Production=yes
EtteMode=2
EtteGraphQLPlayGround=yes
ConcurrencyFactor=1
BlockConfirmations=0
BlockRange=1000
TimeRange=21600
SnapshotFile=snapshot.bin
Thank you
first of all, i would like to thank you for making ette. just like you i've been searching for a solution to query and subscribe to realtime data. thanks again
below is the error messages i'm getting right after i ran ./ette please help me with this
❗️ Failed to fetch block 3110474 : the method eth_getBlockByNumber does not exist/is not available
While trying to set up ette for Avalanche C-Chain, I'm having a problem with it not being able to add blocks to the database due to foreign key constraint being violated. The reason seems to be that the blockhash field in the block's transactions doesn't match the hash field in the block itself. This seems to be happening for all blocks with at least one transaction.
This is the JSON we get from the blockchain API endpoint for an example block:
{
"jsonrpc": "2.0",
"id": 300,
"result": {
"difficulty": "0x1",
"extraData": "0xd683010916846765746886676f312e3133856c696e7578820a2c7241a1364516998c572fd99cc06ca7f4bfcb10febe2ed3019ce17fc702",
"gasLimit": "0x458ecfe",
"gasUsed": "0xaa79",
"hash": "0x4aa30d54066664bbd8f06c39d1dc3e221df24b34d6f70f13ff70a9a602b9b386",
"logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
"miner": "0x0100000000000000000000000000000000000000",
"mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
"nonce": "0x0000000000000000",
"number": "0x143",
"parentHash": "0x709b92a478eac8b472122343766c1f27ae279fb19af1ca17e8601c198675bc8f",
"receiptsRoot": "0x37de9c18b3cdadd7066c737c0a353fa86209d50bc1d565747a636a80140ec114",
"sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
"size": "0x2e9",
"stateRoot": "0x8dbecfa700e97d335ff4f743592ae29078e1c48b50f9f2942cd6b58bad8934a3",
"timestamp": "0x5f89bfe8",
"totalDifficulty": "0x143",
"transactions": [{
"blockHash": "0x4aa30d54066664bbd8f06c39d1dc3e221df24b34d6f70f13ff70a9a602b9b386",
"blockNumber": "0x143",
"from": "0x4abef613822fb2031d897e792f89c896ddafc466",
"gas": "0xaa79",
"gasPrice": "0x6d6e2edc00",
"hash": "0xc98a2b320fc215202faf8a6fbd67a8e58ded4320234979f6b2da1cec97f9dc53",
"input": "0xa8cbe8850d892b7a4f1dc7edce9854d541db0cb91346f0d3615b2eebfae65b2c6eb93d96",
"nonce": "0x18",
"to": "0x448624cf94a47e34a1a694296c0de7b6f872be1d",
"transactionIndex": "0x0",
"value": "0x0",
"v": "0x150f7",
"r": "0x91daf0079e38117227be4a1862beb3739654f647dc1c5b99d40121aed08d93a1",
"s": "0x3fe81b66da434b165180306c44d319008ec664eff82b4524de5bebb7d4126d6c"
}],
"transactionsRoot": "0xaa8b37f5b3d4700ee10602effeea5305088ce9c2cf817867e72b1f0c6efbcb67",
"uncles": []
}
}
And this is a dump of the same block when ette attempts to store it in the database:
{
"Block": {
"Hash": "0xf81dbb7730c2f40cb39f382d9abd72fcd4fdae50bea86f126c95197a81ec6ec9",
"Number": 323,
"Time": 1602863080,
"ParentHash": "0x709b92a478eac8b472122343766c1f27ae279fb19af1ca17e8601c198675bc8f",
"Difficulty": "1",
"GasUsed": 43641,
"GasLimit": 72936702,
"Nonce": "0x0",
"Miner": "0x0100000000000000000000000000000000000000",
"Size": 710,
"StateRootHash": "0x8dbecfa700e97d335ff4f743592ae29078e1c48b50f9f2942cd6b58bad8934a3",
"UncleHash": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
"TransactionRootHash": "0xaa8b37f5b3d4700ee10602effeea5305088ce9c2cf817867e72b1f0c6efbcb67",
"ReceiptRootHash": "0x37de9c18b3cdadd7066c737c0a353fa86209d50bc1d565747a636a80140ec114",
"ExtraData": "1oMBCRaEZ2V0aIZnbzEuMTOFbGludXiCCixyQaE2RRaZjFcv2ZzAbKf0v8sQ/r4u0wGc4X/HAg==",
"Transactions": {
"Hash": "",
"From": "",
"To": "",
"Contract": "",
"Value": "",
"Data": null,
"Gas": 0,
"GasPrice": "",
"Cost": "",
"Nonce": 0,
"State": 0,
"BlockHash": "",
"Events": {
"BlockHash": "",
"Index": 0,
"Origin": "",
"Topics": null,
"Data": null,
"TransactionHash": ""
}
},
"Events": {
"BlockHash": "",
"Index": 0,
"Origin": "",
"Topics": null,
"Data": null,
"TransactionHash": ""
}
},
"Transactions": [
{
"Tx": {
"Hash": "0xc98a2b320fc215202faf8a6fbd67a8e58ded4320234979f6b2da1cec97f9dc53",
"From": "0x0000000000000000000000000000000000000000",
"To": "0x448624CF94a47E34a1A694296c0De7B6f872Be1D",
"Contract": "",
"Value": "0",
"Data": "qMvohQ2JK3pPHcftzphU1UHbDLkTRvDTYVsu6/rmWyxuuT2W",
"Gas": 43641,
"GasPrice": "470000000000",
"Cost": "20511270000000000",
"Nonce": 24,
"State": 1,
"BlockHash": "0x4aa30d54066664bbd8f06c39d1dc3e221df24b34d6f70f13ff70a9a602b9b386",
"Events": {
"BlockHash": "",
"Index": 0,
"Origin": "",
"Topics": null,
"Data": null,
"TransactionHash": ""
}
},
"Events": []
}
]
}
Two changes I've noticed are the hash field of the block (note that all the other block fields containing hashes are consistent with the source JSON) and the from field of the transaction (I haven't looked into this part yet though). The changed block hash (0xf81dbb7730c2f40cb39f382d9abd72fcd4fdae50bea86f126c95197a81ec6ec9) doesn't exist in the blockchain.
I tried current main branch and the 1.6.0 release in case it was a regression bug, but in both cases I've seen the same Postgres errors. Any suggestions on where to look further trying to pinpoint it down?
When quering a contract and trying to find out when and how much eth was actually sent, we cannot only look at the to and the value field of the transactions in the block, but we must also check the internal transactions triggered.
Do you plan to support them as well? (using https://openethereum.github.io/JSONRPC-trace-module#trace_replayblocktransactions would also enable you to put more data in the, like adding the reason for a tx failure)
This one might need a little investigating: sometimes ette receives GOAWAY from the node and handles that as any type of error. An example log message follows:
Post "https://mainnet.infura.io/v3/x": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
IMO ette shouldn't handle response errors generically, this might be the real issue here.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.