Problem Description:
At present, the pooler functionality exhibits optimal performance only when employed with larger epoch sizes, owing to the substantial computational load involved in aggregate calculations. Utilizing the pooler with an epoch size set to 1 triggers a domino effect of issues, commencing with the failure of the 24-hour trade volume aggregate computation, which consequently disrupts the entire process.
Proposed Solution:
The root cause of this failure can be attributed to the fact that the calculation of the 24-hour trade volume aggregate relies heavily on prior finalized epoch data in order to perform efficient computations (derived from the previous epoch). However, the time span of 12 seconds
(as observed with ETH), within which the last epoch is expected to finalize all operations, is inadequate. As a result, for each epoch, a complete recalculation from scratch is attempted. This entails retrieving an extensive dataset either from the RPC or the local Cache, contingent on data availability in the cache.
Consequently, an excessive volume of RPC requests is generated, leading to a surge in Rate Limit errors. This, in turn, triggers a chain reaction of failures, rendering the system entirely non-functional.
To address this, a revamped approach is recommended for the computation of the 24-hour trade volume aggregate. This new approach should eliminate the dependence on finalized calculations and instead harness the locally stored base snapshots submitted by the snapshotter for the most recent unfinalized epochs. Such a redesign will significantly enhance the system's overall efficiency, enabling seamless support for an epoch size of 1.
This enhancement holds the promise of resolving the existing challenges and significantly improving the functionality of the pooler across different epoch sizes.