Ensure predictable latency to apply chunks about nearcore HOT 3 CLOSED

Here is the VM Instance: https://console.cloud.google.com/compute/instancesDetail/zones/europe-west4-a/instances/crt-mainnet?hl=en&project=nearone-crt. It is using this terraform script: https://github.com/Near-One/infra-ops/pull/64

It is currently downloading blocks

Apr 24 13:00:13 crt-mainnet neard[6014]: 2024-04-24T13:00:13.293735Z  INFO stats: #117520067 Downloading blocks 17.79% (2519 left; at 117520067) 32 peers ⬇ 10.0 MB/s ⬆ 4.23 MB/s 1.00 bps 958 Tgas/s CPU: 472%, Mem: 9.67 GB

I believe it should automatically appear on nearone Grafana, I'll share the link when it does.

from nearcore.

The node has been provisioned and is now running, here are the relevant Grafana dashboards:

• Chunk apply latency: https://nearone.grafana.net/goto/VZp4kpfSR?orgId=1
• Blockchain utilization: https://nearone.grafana.net/goto/MHO7kpfSR?orgId=1

I already see some long chunk apply latencies on shards 2, 3 and 5.

I think we can start with 2s threshold for alert, I see 2 such events on shard 5 in the last 6 hours. I'll add some logging to the node to find out the problematic block heights.

from nearcore.

We decided to decommission the node and instead focus on the FT benchmark, so closing this issue.

from nearcore.