Solves issue #12

The current list of pair addresses used in UniswapV2, Sushiswap, and Quickswap are outdated and include inactive or low-activity pairs.
To enhance the user experience and data accuracy, we need to curate and create a new list of top pairs for all three platforms.

Related to PowerLoom/pooler#8

Is your feature request related to a problem?
The present architecture of the indexer and aggregator services that generate important datapoints for the Uniswap v2 use case does not deal well with broken chains and skipped snapshots that can arise because of

• network errors
• cache errors
• RPC timeouts or rate limits
  ...among many other factors that can go wrong.

Its attempt to find a common end epoch across 100+ projects to just get started on building aggregates along with a lot of convoluted logic on trying to sync up other higher order aggregates with the same causes the entire process to come to a halt in case of above issues creating gaps or duplicate entries in snapshot chains. The end result is that even though snapshots continue to proceed fine, there are no usable data points generated or updated.

Describe the solution you'd like

Move the indexing and aggregation ahead as long as majority of the snapshotted projects' snapshot chains progress fine. A couple of projects with broken chains should not cause an entire cessation in data point generation and updation.

Describe alternatives you've considered

Not applicable

Additional context

This scope is only for the phase where these nodes participate in an offchain consensus system and maintain a local state of the DAG chain and other indexes and aggregates on top of it. In the next phase of release, the protocol state of DAG chains, indexes, aggregates will be available on its own protocol chain and breakages in snapshot chains will not be a matter of concern.

Apart from the logical issue itself of halted indexing and aggregation, there are violations of accepted Pythonic design patterns, coding styles and standards which need to be addressed.

Top level issue to track State Builder Development

State Builder agent has the following responsibilities

• Build Snapshot DAG chains
• Build Aggregate DAG chains
• Segment and Store data on IPFS

Current State and Todo

Initially to be run by powerloom but later will be completely decentralized

• Snapshot DAG Chain finalizer (sequencer component) (todo)
• Aggregate DAG Chain finalizer (sequencer component) (todo)
• Design Architecture (in progress)
• Dag chain won't move forward if data is missing. If data is missing it will interact with the contract to release a FixEvent and wait for missing data to be populated and then catch up with everything else
• Pruning and Segmentation (Can be GO based, @swagftw can update audit protocol services)
• Dag Chain Verifier (Update and redesign to fit new decentralized architecture) (todo)

Describe the bug

Web3 storage token is not getting populated properly in generated settings.json while running build.sh from deploy causing dag-status-reporter to crash.

To Reproduce

Affected versions: current dockerify build

Steps to reproduce the behavior:

1. Run ./build.sh with dockerify tag in deploy
2. Check generated settings.json file

Expected behavior
Web3 storage token should get populated properly from .env file in deploy

Describe the bug

• Service seems to be getting stuck due to unknown reason
• Code clean up is required

Expected behavior

• Continuous token aggregation

Proposed Solution

• Code review is necessary along with code cleanup

Caveats
Not sure yet

Additional context
NA

Is your feature request related to a problem?
To complete the feature request on snapshotter implementations of Powerloom Protocol to expose an internal API for snapshot processing status per epoch PowerLoom/pooler#40, it is necessary that the payload commit service captures this information in a transient cache entry which can be parsed to generate a snapshot's status report on its state transitions.

Describe the solution you'd like
When the payload commit service of audit protocol submits a snapshot against a project ID to the relayer service, capture its success or failure from the response returned.

The proposed datamodel is as follows

type SnapshotterStateUpdate struct {
	Status    string                 `json:"status"`
	Error     string                 `json:"error"`
	Extra     map[string]interface{} `json:"extra"`
	Timestamp int64                  `json:"timestamp"`
}

This is to be recorded against the state transition of REALYER_SEND as detailed in PowerLoom/pooler#40

Describe alternatives you've considered
NA

Additional context
NA

Describe the bug

1. If a project ID is registered for building indexes on, and there have been no snapshots received on it nor DAG chains built, it can cause the index generation (eg. 24 hour, 7 day timeseries) to completely fail for all projects registered for indexing.

The individual tasks launched as below may return None because of the decorator @acquire_bounded_semaphore applied to the function get_epoch_end_per_project(),

which will cause the following to fail

2. Aggregation for Uniswap specific data point, trade volume across a segment of the DAG chain of snapshots can fail because of a few DAG blocks that do not have a snapshot payload encapsulated in them.

6|ap-proto-indexer  | ERROR    pair_data_aggregation_service 2023-03-23 14:41:03,624 624 pair_data_aggregation_service-process_pairs_trade_volume_and_reserves: Error in process_pairs_trade_volume_and_reserves: 'NoneType' object is not subscriptable
6|ap-proto-indexer  | Traceback (most recent call last):
6|ap-proto-indexer  |   File "/src/pair_data_aggregation_service.py", line 498, in process_pairs_trade_volume_and_reserves
6|ap-proto-indexer  |     pair_trade_volume_7d = calculate_pair_trade_volume(dag_chain_7d)
6|ap-proto-indexer  |   File "/src/pair_data_aggregation_service.py", line 288, in calculate_pair_trade_volume
6|ap-proto-indexer  |     pair_trade_volume.total_volume = sum(map(lambda x: x['data']['payload']['totalTrade'], dag_chain))
6|ap-proto-indexer  |   File "/src/pair_data_aggregation_service.py", line 288, in <lambda>
6|ap-proto-indexer  |     pair_trade_volume.total_volume = sum(map(lambda x: x['data']['payload']['totalTrade'], dag_chain))
6|ap-proto-indexer  | TypeError: 'NoneType' object is not subscriptable

To Reproduce

As described in the previous section, the bug arises when the appropriate conditions are (un)satisfied.

Expected behavior

Errors concerning the state of individual project IDs should not halt aggregation generation on a collection of such project IDs.

Proposed Solution

1. Handle specific exception from wrapped function in @acquire_bounded_semaphore to ensure a blanket None is not returned in case of all exceptions
2. Ignore DAG blocks with null payloads while calculating aggregated trade volume.

Caveats
No caveats. These changes allow unstable or testing systems to continue generating aggregate data points and not have the need to reset local state of DAG chains and indexes built on them.

Additional context
Independent of OS and Python version, this will show up as long as the conditions described above are encountered.

Is your feature request related to a problem?
Presently, payload commit service consumes large payloads over RabbitMQ from snapshotters to ultimately upload the contents to IPFS and/or web3 storage. This is a huge overhead on system resources and parallel work is underway on the snapshotter repos to integrate these functionalities natively in there.

Describe the solution you'd like
Do not consume entire snapshot contents as payload over RabbitMQ and instead only act as a dispatcher for the snapshot submission transaction to relayer.

Describe alternatives you've considered
NA

Additional context
NA

Is your feature request related to a problem?

The present, off chain version of audit protocol maintains localized DAG chains of finalized snapshots.

The payload commit service specifically sets a state marker, project_first_epoch_end_height that will affect the epochs to be queried from the DAG finalizer service when it performs a self healing of its DAG chain of snapshots. This will give rise to indeterminate behavior in peers that have not joined a consensus network "from the beginning" because of the following code in DAG finalizer.

When a peer is 'late' to the network, its project_first_epoch_end_height will depend upon the first snapshot submission received by the local payload commit service. This will obviously vary across peers joining in at different times, which will cause a completely different epoch to be queried on the consensus service against a missing block height entry in the DAG chain.

Describe the solution you'd like

• Move the project_first_epoch_end_height state marker to off chain consensus service
  AND/OR
• Before a late peer begins submitting snapshot, preload a snapshot state as described following(which includes project_first_epoch_end_height) and let it catch up on the entirety of the chain.

Describe alternatives you've considered

Not applicable

Additional context

Not applicable

This is issue is for improving and eventually fixing Pruning-Archival service written in go.

This thread can be used for continuous discussion and progress of the changes required for this service to work properly.
Flow diagram for the existing service can be found here

• need to understand flaws and gaps in current flow and compare with expected behavior and improve
• update the service with updated behaviour

Current Behaviour:

• service polls redis cache after configured time and runs pruning and archival on every project
• segments can be created anytime and after uncertain duration hence polling can be bad
• unclear code flow and hard to read
• lacks proper retry mechanism
• does not seem to prune & archive DAG project chains consistently.

Expected Behaviour:

• Should prune DAG chains and archive and maintain valid state.
• Remove need of polling
• Should be treated as worker on tasks appended to message queue
• Way to test locally
• Readable and modular code

Do you want to work on this issue?

Yes