Exec Layer node getLogs requests stop after EL restart about nimbus-eth2 HOT 6 OPEN

This node is outdated and must be updated for Nimbus >= v24.2.1 before Deneb (5 days deadline).

from nimbus-eth2.

Good point. We just finished upgrading EL nodes, will do BN next.

from nimbus-eth2.

Searching the logs for Failed to obtain the latest block from the EL is only logged one time. After that, there are still periodic syncEth1Chain tick that start new eth_getBlockByNumber('latest', false) requests, but these subsequent ones never seem to finish / time out / fail.

    debug "syncEth1Chain tick"

    if bnStatus == BeaconNodeStatus.Stopping:
      await m.stop()
      return

    if m.eth1Chain.hasConsensusViolation:
      raise newException(CorruptDataProvider, "Eth1 chain contradicts Eth2 consensus")

    let latestBlock = try:
      raiseIfNil connection.trackedRequestWithTimeout(
        "getBlockByNumber",
        rpcClient.eth_getBlockByNumber(blockId("latest"), false),
        web3RequestsTimeout)
    except CatchableError as err:
      warn "Failed to obtain the latest block from the EL", err = err.msg
      raise err

• bnStatus == BeaconNodeStatus.Stopping cannot be because that is only set on exit
• m.eth1Chain.hasConsensusViolation cannot be because the locations that set it to true produce a unique log line
• connection.trackedRequestWithTimeout is still repeatedly hit, as seen by Sending message to RPC server log line immediately after each syncEth1Chain tick log

Maybe something weird in trackedRequestWithTimeout related to or that makes it get stuck:

• status-im/nim-chronos#516

from nimbus-eth2.

Do you have eth1_latest_head metric from that time as well? And also eth1_synced_head

from nimbus-eth2.

Extended the logs here as well to have a better understanding what's going on:

• #6049

from nimbus-eth2.

Do you have eth1_latest_head metric from that time as well?

Not on the dashboard but it can be generated:

And also eth1_synced_head

from nimbus-eth2.