Reducing state witness size - compression about nearcore HOT 8 CLOSED

@walnut-the-cat yes, #10566 is pretty much the same as "Caching large state values" but with a slightly different approach

from nearcore.

In that case, I will merge #10566 into this one. Feel free to re-open if you think otherwise

from nearcore.

Thanks for the write up-! couple questions

1. Could you share how I can read dashboard better?
2. What does 'cache_state_values' in witness size graphs mean? Does it mean size of state witness when 'we don't do any compression but enable state value caching on validator side'?
3. How long does 'decompression' take? Do we expect similar amount time to compression time? (thus 2x compression time latency)

from nearcore.

@walnut-the-cat

1. I've added more detailed description for that dashboard
2. Yes, it means caching state values with no compression
3. decompression time is already included, added that to the description

from nearcore.

Should we consider this task (#10566) as overlap of what you are working on?

from nearcore.

Top N values as part of the Protocol

It was mentioned that we would globally cache to top N contracts based on usage. A good metric to use here can be size_of_contract * num_times_called_in_prev_epoch? This would provide the maximum size benefits assuming the load of the current epoch would be similar to the previous epoch.

from nearcore.

@shreyan-gupta size_of_contract * num_times_called_in_prev_epoch makes sense if we have a lot of cache churn. In my experiments with N=1000 per each shard it didn't fill even half-way in 6 days: dashboard. So it seems like straight-forward LRU would work perfectly fine.

from nearcore.

Compression was implemented in #10715
Closing this for now, a new issue will be created for the long-term (Top N values as part of the Protocol) solution

from nearcore.