harmony-one / explorer-v2-backend Goto Github PK

Proxy contract implementing the transparent, uups, and beacon proxy standards that proxy to token standards (20, 721, 1155) are not be processed by the indexer. As such tokens are not showing up in the explorer list. (e.g. https://explorer.harmony.one/tx/0x072f0f568990925ff294c6ade9137c6d1776c64ba5621f3af17e9cbe7eed5cf4)

Proposed solution:
1 how to track proxy:
we can check each contract detected to determine if it is a proxy (openzeppelin/upgrades-core has functions for proxy detection)
(contractAddressIndexer)

2 how to track transfers
transfers check through events fired by the contract
consider sync function that we can call as part of the cli if we have out of sync issues

3 how to track state change (between upgrades)

• see (4)

4 ideas how to recalculate explorer storage (is it even needed)

• we can revalidate all accounts (slow)
• mark in the browser that contract last upgrade status for user to manually check their balances
• detect upgrade and check the before and after state of the contract (has the internal variables changed, any new variables added, etc) - https://medium.com/@eiki1212/ethereum-state-trie-architecture-explained-a30237009d4e

Parent ticket is #33. Please refer to parent ticket for the yml file.
This ticket refers to the logs api group and its endpoints. It is considered complete when all the api endpoints below are implemented

• /logs/getLogs: Get "Internal Transactions" by Block Range

use one instance of response Promise for latest blocks and latest transactions requests instead of many parallel requests before the response gets cached

During the last outage, the indexer was stuck and no indicator of such was available.

The proposal is to implement a new flag isStuck with the following condition :

• true when the block is the same for 1m
• false otherwise

During a recent outage, the indexer status page showed

"state":{"lastSyncedBlocks":[{"shardID":0,"blockNumber":28475983,"latestBlockchainBlock":28475983,"isSynced":true}]}

the issue is that the indexer was stuck at block 28475983 and wasn't able to fetch the latest network block.

We need to understand, why the latest block fetch number didn't work (RPC ? Indexer backend code issue ?)

When an indexer is configured to index multiple shard at once, the initialization is done in parallel. Howver if one RPC is not working, the entire indexer doesn't start at all.

it's best to

1. display an error message when RPC initialization doesn't work
2. move on and don't index the shard covered by faulty RPC

I am having an issue with the following RPC call to wss://ws.s0.t.hmny.io :

[WebSocketRPC:wss://ws.s0.t.hmny.io] [2021-12-12T11:09:21.014Z] Call error {
method: 'trace_block',
params: [ '0xff53e' ],
err: {
code: -32601,
message: 'the method trace_block does not exist/is not available'
}
}

Is there some kind of an error I am making, or is the 'trace_block' unavailable for some reason?

I started indexing from block 0 and currently stuck at block 990419. I am happy to provide more information if needed. Any ideas?

Thanks.

Original message from discord: link

Description
Some internal transactions are missing in the database.
Example: no internal transactions for tx 0xc38346a5802e8a4733ea8f71d2b4ea87946312cb6f98789c32d581e7c83c94f4

How to reproduce
No steps at this time.
If re-sync indexer from the same block (21875003 for tx above), internal transactions will be successfully parsed from traced block and written to database.

It's is quite strange for me the way the Circulating Supply is being calculated for an ERC20 token.
Basically it seems to not consider balances that are owned by contracts. Is that it?

Is it possible for it to be customizable by the team developing the ERC20 token?

Hi,

Quick question about the explorer v2. I have an app that is using the ethers library to access transaction histories through the etherscan provider on ethereum (https://docs.ethers.io/v5/api/providers/api-providers/#EtherscanProvider). Will the harmony explorer v2 support this? Thanks!

Hello, could I check if filters are supported for getRelatedTransactionsByType query? Specifically, filtering by block number so that I only received transactions >= a certain block number.

From what I've read in the code, the filters passed in as parameters are not utilised at the moment

Parent ticket is #33. Please refer to parent ticket for the yml file.
This ticket refers to the gas tracker api group and its endpoints. It is considered complete when all the api endpoints below are implemented

• /gastracker/gasestimate: Get Estimation of Confirmation Time
• /gastracker/gasoracle: Get Gas Oracle

https://explorer.harmony.one/tx/0x44a4e2099dc8bd5cc170b4cab7678d61f3751389edc4d70e7260557c0b4cf36f?activeTab=0

curl -d '{ "jsonrpc":"2.0", "method":"eth_getTransactionByHash", "params":[ "0x44a4e2099dc8bd5cc170b4cab7678d61f3751389edc4d70e7260557c0b4cf36f" ], "id":1 }' -H 'Content-Type:application/json' -X POST 'https://api.s0.t.hmny.io'

the transaction has internal ONE transfer, which is shown neither in tables nor on details page

Parent ticket is #33. Please refer to parent ticket for the yml file.
This ticket refers to the Proxy api group and its endpoints. It is considered complete when all the api endpoints below are implemented

• /proxy/eth_blockNumber: eth_blockNumber
• /proxy/eth_getBlockByNumber: Returns information about a block by block number.
• /proxy/eth_getUncleByBlockNumberAndIndex: Returns information about a block by block number.
• /proxy/eth_getBlockTransactionCountByNumber: Returns the number of transactions in a block.
• /proxy/eth_getTransactionByHash: Returns the information about a transaction requested by transaction hash.
• /proxy/eth_getTransactionByBlockNumberAndIndex: Returns information about a transaction by block number and transaction index position.
• /proxy/eth_getTransactionCount: Returns the number of transactions performed by an address.
• /proxy/eth_sendRawTransaction: Submits a pre-signed transaction for broadcast to the Ethereum network.
• /proxy/eth_getTransactionReceipt: Returns the receipt of a transaction by transaction hash.
• /proxy/eth_call: Executes a new message call immediately without creating a transaction on the block chain.
• /proxy/eth_getCode: Returns code at a given address.
• /proxy/eth_getStorageAt: Returns the value from a storage position at a given address.
• /proxy/eth_gasPrice: Returns the current price per gas in wei.
• /proxy/eth_estimateGas: Makes a call or transaction, which won't be added to the blockchain and returns the used gas.

This task involves documenting the api that are currently available.

This will feed into subsequent tasks for the backend with similar titles.

Let's display total transactions in the transaction table. See etherscan example below

Parent ticket is #33. Please refer to parent ticket for the yml file.
This ticket refers to the transactions api group and its endpoints. It is considered complete when all the api endpoints below are implemented

• /transaction/getstatus: Check Contract Execution Status
• /transaction/gettxreceiptstatus: Check Transaction Receipt Status

Parent ticket is #33. Please refer to parent ticket for the yml file.
This ticket refers to the stats api group and its endpoints. It is considered complete when all the api endpoints below are implemented

• /stats/ethsupply: Get Total Supply of Ether (in our case Harmony ONE tokens)
• /stats/ethprice: Get Ether Last Price (in our case this is the Harmony token price)
• /stats/chainsize: Get Harmony Nodes Size
• /stats/nodecount: Get Total Nodes Count

@ArtemKolodko could you look at the below issue ?

[PostgresStorage:shard1] [2023-05-25T08:00:25.837Z] Removing internal transactions where block number < 44642856
node:events:368
      throw er; // Unhandled 'error' event
      ^

Error: Connection terminated unexpectedly
    at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:509:28)
    at Connection.emit (node:events:390:28)
    at Connection.emit (node:domain:475:12)
    at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:58:12)
    at Socket.emit (node:events:390:28)
    at Socket.emit (node:domain:475:12)
    at TCP.<anonymous> (node:net:687:12)
Emitted 'error' event on BoundPool instance at:
    at Client.idleListener (/usr/src/app/node_modules/pg-pool/index.js:57:10)
    at Client.emit (node:events:390:28)
    at Client.emit (node:domain:475:12)
    at Client._handleErrorEvent (/usr/src/app/node_modules/pg/lib/client.js:319:10)
    at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:149:16)
    at Object.onceWrapper (node:events:509:28)
    [... lines matching original stack trace ...]
    at TCP.<anonymous> (node:net:687:12) {
  client: Client {
    _events: [Object: null prototype] { error: [Function (anonymous)] },
    _eventsCount: 1,
    _maxListeners: undefined,
    connectionParameters: ConnectionParameters {
      user: 'postgres',
      database: 's1',
      port: 5432,
      host: 'harmony-shard1-2-3.cluster-clkxvqwosgiw.us-east-1.rds.amazonaws.com',
      binary: false,
      options: undefined,
      ssl: false,
      client_encoding: '',
      replication: undefined,
      isDomainSocket: false,
      application_name: undefined,
      fallback_application_name: undefined,
      statement_timeout: false,
      idle_in_transaction_session_timeout: false,
      query_timeout: false,
      connect_timeout: 0
    },
    user: 'postgres',
    database: 's1',
    port: 5432,
    host: 'harmony-shard1-2-3.cluster-clkxvqwosgiw.us-east-1.rds.amazonaws.com',
    replication: undefined,
    _Promise: [Function: Promise],
    _types: TypeOverrides {
      _types: {
        getTypeParser: [Function: getTypeParser],
        setTypeParser: [Function: setTypeParser],
        arrayParser: { create: [Function: create] },
        builtins: {
          BOOL: 16,
          BYTEA: 17,
          CHAR: 18,
          INT8: 20,
          INT2: 21,
          INT4: 23,
          REGPROC: 24,
          TEXT: 25,
          OID: 26,
          TID: 27,
          XID: 28,
          CID: 29,
          JSON: 114,
          XML: 142,
          PG_NODE_TREE: 194,
          SMGR: 210,
          PATH: 602,
          POLYGON: 604,
          CIDR: 650,
          FLOAT4: 700,
          FLOAT8: 701,
          ABSTIME: 702,
          RELTIME: 703,
          TINTERVAL: 704,
          CIRCLE: 718,
          MACADDR8: 774,
          MONEY: 790,
          MACADDR: 829,
          INET: 869,
          ACLITEM: 1033,
          BPCHAR: 1042,
          VARCHAR: 1043,
          DATE: 1082,
          TIME: 1083,
          TIMESTAMP: 1114,
          TIMESTAMPTZ: 1184,
          INTERVAL: 1186,
          TIMETZ: 1266,
          BIT: 1560,
          VARBIT: 1562,
          NUMERIC: 1700,
          REFCURSOR: 1790,
          REGPROCEDURE: 2202,
          REGOPER: 2203,
          REGOPERATOR: 2204,
          REGCLASS: 2205,
          REGTYPE: 2206,
          UUID: 2950,
          TXID_SNAPSHOT: 2970,
          PG_LSN: 3220,
          PG_NDISTINCT: 3361,
          PG_DEPENDENCIES: 3402,
          TSVECTOR: 3614,
          TSQUERY: 3615,
          GTSVECTOR: 3642,
          REGCONFIG: 3734,
          REGDICTIONARY: 3769,
          JSONB: 3802,
          REGNAMESPACE: 4089,
          REGROLE: 4096
        }
      },
      text: {},
      binary: {}
    },
    _ending: true,
    _connecting: false,
    _connected: true,
    _connectionError: false,
    _queryable: false,
    connection: Connection {
      _events: [Object: null prototype] {
        newListener: [Function (anonymous)],
        connect: [Function (anonymous)],
        sslconnect: [Function (anonymous)],
        authenticationCleartextPassword: [Function: bound _handleAuthCleartextPassword],
        authenticationMD5Password: [Function: bound _handleAuthMD5Password],
        authenticationSASL: [Function: bound _handleAuthSASL],
        authenticationSASLContinue: [Function: bound _handleAuthSASLContinue],
        authenticationSASLFinal: [Function: bound _handleAuthSASLFinal],
        backendKeyData: [Function: bound _handleBackendKeyData],
        error: [Function: bound _handleErrorEvent],
        errorMessage: [Function: bound _handleErrorMessage],
        readyForQuery: [Function: bound _handleReadyForQuery],
        notice: [Function: bound _handleNotice],
        rowDescription: [Function: bound _handleRowDescription],
        dataRow: [Function: bound _handleDataRow],
        portalSuspended: [Function: bound _handlePortalSuspended],
        emptyQuery: [Function: bound _handleEmptyQuery],
        commandComplete: [Function: bound _handleCommandComplete],
        parseComplete: [Function: bound _handleParseComplete],
        copyInResponse: [Function: bound _handleCopyInResponse],
        copyData: [Function: bound _handleCopyData],
        notification: [Function: bound _handleNotification],
        end: [Function: bound onceWrapper] {
          listener: [Function (anonymous)]
        }
      },
      _eventsCount: 23,
      _maxListeners: undefined,
      stream: Socket {
        connecting: false,
        _hadError: false,
        _parent: null,
        _host: 'harmony-shard1-2-3.cluster-clkxvqwosgiw.us-east-1.rds.amazonaws.com',
        _readableState: ReadableState {
          objectMode: false,
          highWaterMark: 16384,
          buffer: BufferList { head: null, tail: null, length: 0 },
          length: 0,
          pipes: [],
          flowing: true,
          ended: false,
          endEmitted: false,
          reading: true,
          constructed: true,
          sync: false,
          needReadable: true,
          emittedReadable: false,
          readableListening: false,
          resumeScheduled: false,
          errorEmitted: true,
          emitClose: false,
          autoDestroy: true,
          destroyed: true,
          errored: Error: read ECONNRESET
              at TCP.onStreamRead (node:internal/stream_base_commons:220:20) {
            errno: -104,
            code: 'ECONNRESET',
            syscall: 'read'
          },
          closed: true,
          closeEmitted: true,
          defaultEncoding: 'utf8',
          awaitDrainWriters: null,
          multiAwaitDrain: false,
          readingMore: false,
          dataEmitted: true,
          decoder: null,
          encoding: null,
          [Symbol(kPaused)]: false
        },
        _events: [Object: null prototype] {
          end: [
            [Function: onReadableStreamEnd],
            [Function (anonymous)],
            [Function (anonymous)]
          ],
          error: [Function: reportStreamError],
          close: [Function (anonymous)],
          data: [Function (anonymous)]
        },
        _eventsCount: 4,
        _maxListeners: undefined,
        _writableState: WritableState {
          objectMode: false,
          highWaterMark: 16384,
          finalCalled: false,
          needDrain: false,
          ending: false,
          ended: false,
          finished: false,
          destroyed: true,
          decodeStrings: false,
          defaultEncoding: 'utf8',
          length: 0,
          writing: false,
          corked: 0,
          sync: false,
          bufferProcessing: false,
          onwrite: [Function: bound onwrite],
          writecb: null,
          writelen: 0,
          afterWriteTickInfo: null,
          buffered: [],
          bufferedIndex: 0,
          allBuffers: true,
          allNoop: true,
          pendingcb: 0,
          constructed: true,
          prefinished: false,
          errorEmitted: true,
          emitClose: false,
          autoDestroy: true,
          errored: Error: read ECONNRESET
              at TCP.onStreamRead (node:internal/stream_base_commons:220:20) {
            errno: -104,
            code: 'ECONNRESET',
            syscall: 'read'
          },
          closed: true,
          closeEmitted: true,
          [Symbol(kOnFinished)]: []
        },
        allowHalfOpen: false,
        _sockname: null,
        _pendingData: null,
        _pendingEncoding: '',
        server: null,
        _server: null,
        [Symbol(async_id_symbol)]: 168740,
        [Symbol(kHandle)]: null,
        [Symbol(kSetNoDelay)]: true,
        [Symbol(lastWriteQueueSize)]: 0,
        [Symbol(timeout)]: null,
        [Symbol(kBuffer)]: null,
        [Symbol(kBufferCb)]: null,
        [Symbol(kBufferGen)]: null,
        [Symbol(kCapture)]: false,
        [Symbol(kBytesRead)]: 604800,
        [Symbol(kBytesWritten)]: 1032744
      },
      _keepAlive: false,
      _keepAliveInitialDelayMillis: 0,
      lastBuffer: false,
      parsedStatements: {},
      ssl: false,
      _ending: true,
      _emitMessage: false,
      _connecting: true,
      [Symbol(kCapture)]: false
    },
    queryQueue: [],
    binary: false,
    processID: 1175,
    secretKey: -1977370399,
    ssl: false,
    _connectionTimeoutMillis: 0,
    _connectionCallback: null,
    release: [Function (anonymous)],
    activeQuery: null,
    readyForQuery: true,
    hasExecuted: true,
    _poolUseCount: 6144,
    [Symbol(kCapture)]: false
  }
}
error Command failed with exit code 1.

on the HRC20/721/1155 token page, there should be a count of Holders, and a 'Holders' section under the transaction table, just like etherscan

Parent ticket is #33. Please refer to parent ticket for the yml file.
This ticket refers to the accounts api group and its endpoints. It is considered complete when all the api endpoints below are implemented

• /account/balance: returns the ether balance of a given address
• /account/balanceMulti: Returns the balance of the accounts from a list of addresses.
• /account/txlist: Returns the list of transactions performed by an address, with optional pagination.
• /account/txlistinternal: Get "Internal Transactions" by Block Range
• /account/tokentx: Get a list of 'ERC20 - Token Transfer Events' by Address
• /account/tokennfttx: Get a list of 'ERC721 - Token Transfer Events' by Address
• /account/getminedblocks: Get list of Blocks Mined by Address

The current implementation supports contract indexing only on shard 0.
Need to add support for contract indexing on shards 1-3.

Parent ticket is #33. Please refer to parent ticket for the yml file.
This ticket refers to the Tokens api group and its endpoints. It is considered complete when all the api endpoints below are implemented

• /stats/tokensupply: Get ERC20-Token TotalSupply by ContractAddress
• /account/tokenbalance: Get ERC20-Token Account Balance for TokenContractAddress

Parent ticket is #33. Please refer to parent ticket for the yml file.
This ticket refers to the blocks api group and its endpoints. It is considered complete when all the api endpoints below are implemented

• /block/getblockreward: Get Block And Uncle Rewards by BlockNo
• /block/getblockcountdown: Get Estimated Block Countdown Time by BlockNo
• /block/getblocknobytime: Get Block Number by Timestamp

Hello, I try to run this in dev mode, and get this error. How to solve this, please?

yarn run v1.22.18
$ ts-node -r tsconfig-paths/register src/index.ts
[index] [2022-03-20T19:25:52.751Z] Harmony Explorer v0.1.1. Git commit hash: 1af2190
[index] [2022-03-20T19:25:52.760Z] API is disabled
[indexer/rpc/transport/index] [2022-03-20T19:25:55.124Z] RPC transport: Websocket
[api/controllers/cache/index:cache] [2022-03-20T19:26:00.616Z] LRU cache enabled
[indexer/index] [2022-03-20T19:26:01.690Z] Indexer starting... Shards[0]
[store/postgres/index:shard0] [2022-03-20T19:26:01.730Z] postgres://postgres@localhost:5432/e starting...
[store/postgres/index:shard0] [2022-03-20T19:26:01.730Z] Done
[indexer/rpc/transport/ws/WebSocketRPC:wss://ws.s0.t.hmny.io] [2022-03-20T19:26:03.966Z] Open
[indexer/index] [2022-03-20T19:26:05.094Z] Chain ID set to 1666600000
[indexer/indexer/BlockIndexer:shard0] [2022-03-20T19:26:05.183Z] Created
[indexer/indexer/metrics/walletCount] [2022-03-20T19:26:05.213Z] Wallet counter starting...
[indexer/indexer/metrics/walletCount] [2022-03-20T19:26:05.213Z] Heavy task, make sure you need it enabled
[indexer/indexer/LogIndexer:shard0] [2022-03-20T19:26:05.216Z] Created
[indexer/indexer/contracts/ContractIndexer] [2022-03-20T19:26:05.217Z] Created [erc20, erc721]
[indexer/indexer/contracts/ContractIndexer:erc20] [2022-03-20T19:26:05.218Z] Starting...
[indexer/server/index] [2022-03-20T19:26:05.219Z] Indexer info web server listening at http://localhost:3002
[indexer/indexer/BlockIndexer:shard0] [2022-03-20T19:26:05.229Z] Batch failed. Retrying in 2000ms Cannot read properties of undefined (reading 'last_synced_block_number')
[indexer/indexer/BlockIndexer:shard0] [2022-03-20T19:26:05.231Z] Batch decreased to 900
[indexer/indexer/LogIndexer:shard0] [2022-03-20T19:26:05.348Z] Batch failed. Retrying in 2000ms Cannot read properties of undefined (reading 'last_synced_block_number')
[indexer/indexer/LogIndexer:shard0] [2022-03-20T19:26:05.349Z] Batch decreased to 9
[indexer/indexer/contracts/ContractIndexer:erc20] [2022-03-20T19:26:05.350Z] Batch failed Cannot read properties of undefined (reading 'last_synced_block_number')
[indexer/indexer/contracts/ContractIndexer:erc721] [2022-03-20T19:26:05.350Z] Starting...
[indexer/indexer/contracts/ContractIndexer:erc721] [2022-03-20T19:26:05.351Z] Batch failed Cannot read properties of undefined (reading 'last_synced_block_number')
[indexer/indexer/BlockIndexer:shard0] [2022-03-20T19:26:07.270Z] Batch failed. Retrying in 2000ms Cannot read properties of undefined (reading 'last_synced_block_number')
[indexer/indexer/BlockIndexer:shard0] [2022-03-20T19:26:07.272Z] Batch decreased to 810
[indexer/indexer/LogIndexer:shard0] [2022-03-20T19:26:07.352Z] Batch failed. Retrying in 2000ms Cannot read properties of undefined (reading 'last_synced_block_number')
[indexer/indexer/LogIndexer:shard0] [2022-03-20T19:26:07.353Z] Batch decreased to 8
[indexer/indexer/BlockIndexer:shard0] [2022-03-20T19:26:09.290Z] Batch failed. Retrying in 2000ms Cannot read properties of undefined (reading 'last_synced_block_number')
[indexer/indexer/BlockIndexer:shard0] [2022-03-20T19:26:09.291Z] Batch decreased to 729
[indexer/indexer/LogIndexer:shard0] [2022-03-20T19:26:09.355Z] Batch failed. Retrying in 2000ms Cannot read properties of undefined (reading 'last_synced_block_number')
[indexer/indexer/LogIndexer:shard0] [2022-03-20T19:26:09.356Z] Batch decreased to 7
[indexer/index] [2022-03-20T19:26:10.702Z] Unhandled Rejection at: Promise', [object Promise], 'reason:', TypeError: Cannot read properties of undefined (reading 'last_synced_block_number')