Investigate the effects of decreasing function call gas/compute cost about nearcore HOT 6 OPEN

We mined the necessary data from DataBricks databases together @khorolets (we first did it on DataBricks and then I repeated the same queries on BigQuery). Here is the bottom line for shard 2 only:

• On 18th of March, shard 2 produced 65k blocks with total capacity of 65000PGas and utilized capacity of 26000PGas
• There have been 3.2M function call actions, which only in base costs yield 15000PGas

Based on that, decreasing the function call base cost to 700GGas would result in 12800PGas reduction in usage from 26000PGas to 13200PGas, roughly halving the total gas usage in the shard.

It's tricky to extrapolate from these numbers to the effects on the congestion, but a naive estimate would be that this would double the throughput and cut in half the queuing time at the peak load (assuming the usage pattern remains the same).

Query:

SELECT count(*) as count, sum(gas_limit) as gas_limit, sum(gas_used) as gas_used
FROM `bigquery-public-data.crypto_near_mainnet_us.chunks`
WHERE block_date = "2024-03-18" AND shard_id=2

Result:

[{
  "count": "65149",
  "gas_limit": "6.5149e+19",
  "gas_used": "2.6066742283255353e+19"
}]

Query:

SELECT action_kind, count(*) as count
FROM `bigquery-public-data.crypto_near_mainnet_us.receipt_actions`
WHERE block_date = "2024-03-18"
AND shard_id=2
group by action_kind
LIMIT 10

Result:

[{
  "action_kind": "ADD_KEY",
  "count": "6614"
}, {
  "action_kind": "DELETE_ACCOUNT",
  "count": "12"
}, {
  "action_kind": "TRANSFER",
  "count": "4415908"
}, {
  "action_kind": "DELETE_KEY",
  "count": "2199"
}, {
  "action_kind": "CREATE_ACCOUNT",
  "count": "109"
}, {
  "action_kind": "DEPLOY_CONTRACT",
  "count": "22"
}, {
  "action_kind": "STAKE",
  "count": "756"
}, {
  "action_kind": "FUNCTION_CALL",
  "count": "3281705"
}]

from nearcore.

The first step to answer these questions will be to understand how much gas in the chunk is spent on action_function_call. We had a similar question in the past #8258 and I'll do the investigation for this specific action.

from nearcore.

It looks like this data is currently not collected by Prometheus, so I'll look into using the NEAR Indexer database for the estimate. Unfortunately, Indexer DB has been deprecated and https://github.com/PagodaPlatform/congestion-analysis does not work anymore. There is an alternative BigQuery-based database https://docs.near.org/bos/queryapi/big-query, I'll see if it contains the necessary data.

from nearcore.

Answering the question about chunk production time is much trickier as it heavily depends on the validator hardware.
We collect chunk apply time from some validators, for example here is the P99 graph of chunk apply time for shard 2 for a set of 6 validators (that are also chunk producers):

If we assume that chunk throughput increased by 2x and latency of chunk apply to increase by 2x, we would expect the P99 to reach:

• 2s for 2 validators
• 1s for 4 validators

This will lead to 2 of the validators skipping chunks for 10 minutes every hour. To avoid this, we would need to offset the gas cost change with a 2x performance improvement during function call execution.

from nearcore.

One more idea from @tayfunelmas - we can mirror the traffic from the mainnet to check the effects of changing the gas price. I'll work on this next.

Idea from @bowenwang1996 - study the spikes in P99 metric for validators to see if there are systematic issues and try to replay and profile these blocks.

from nearcore.

Following up on our discussion with @akhi3030, the mainnet validator metrics that we should be looking at to understand how much the performance improvements help and how much we can increase the throughput in congested shards:

Validator chunk processing latency

We have a dashboard showing these latencies across a few validators: https://nearinc.grafana.net/goto/RT5ocLJIR?orgId=1

Specifically for this investigation, we are interested in Shard 2:

• This shard experiences regular congestion with a large number of delayed receipts: https://nearinc.grafana.net/goto/dlRfcL1SR?orgId=1
• The latency of chunk processing across validators is spiking in unison with congestion: https://nearinc.grafana.net/goto/krMs5Y1Sg?orgId=1

After the performance improvements are deployed to these validators, we expect lower P50 and P99 peaks in chunk processing latency. These peaks give us an upper bound of how much we can expect to increase the network's throughput during the congested periods before the chunk processing on these validators starts taking >1s and they start lagging behind the chain head.

The new shard 5 is experiencing similar spikes in latency https://nearinc.grafana.net/goto/3lZ3cY1IR?orgId=1, but no consistent congestion yet https://nearinc.grafana.net/goto/Z2fe5LJIR?orgId=1.

We might also be interested in looking at these metrics on one of our RPC nodes: https://nearinc.grafana.net/goto/27QWtLJSg?orgId=1 as we have more control over the nodes. From my observations, these metrics largely agree with the trends on the validators.

At the moment, P99 latency regularly spikes to 1 second, and P50 latency spikes to 400ms - this signals that any increase in throughput during congested times will bring P99 over 1 second.

from nearcore.