Unresponsive REST API and bursts of logs about nimbus-eth2 HOT 2 CLOSED

On these servers, a request for a state that old will take time to process - likely the client fetching the data times out before a response can be generated (which is what the error says), and the unresponsiveness is simply an outcome of the server being overloaded with these requests..

for the public-facing servers, the health check should probably have a sensitivity of a few minutes, ie failures under 5 minutes are to be expected during times of load/spam

from nimbus-eth2.

Okay, then if you consider this normal behavior for such an overloaded system due to REST API being publicly exposed we can lose this issue. I have already reduced sensitivity of Consul healthchecks anyway for all fleets:

• infra-nimbus#d1d762a6 - nimbus.fluffy: lower sensitivity for consul checks
• infra-nimbus#60dd11a8 - holesky,mainnet,sepolia: lower consul sensitivity

And so now these alerts should not trigger most of the time.

from nimbus-eth2.