Giter Club home page Giter Club logo

node-issue-report-collector's Introduction

Onchain Consensus

Table of Contents

Overview

Overall Architecture Onchain Consensus is part of the Admin Module in the overall architecture. It currently serves the following important roles -

  1. Maintains and releases Epoch depending on chain and use case configuration
  2. Checks and completes consensus (if necessary) by interacting with the Protocol State contract for previously released epochs after the submission window has passed
  3. Provides a set of APIs for metrics and system state statistics where snapshotters can report their issues and overall network health can be monitored

Setup

Onchain Consensus, currently, is only relevant to you if you're a validator. Snapshotters can just use the provided reporting URL in their configuration. But if you're a developer and want to play around with the system and build your use case, then you should follow these instructions to set up the Powerloom System.

Development Instructions

These instructions are needed if you're planning to run the system using build-dev.sh from deploy.

Generate Config

The Onchain Consensus system needs the settings.json file to be present in the settings directory. We've provided settings/settings.example.json for you to get started. Changes are trivial. Copy settings.example.json to settings.json and make the necessary configuration changes.

Configuring settings.json

There are a lot of configurations in the settings.json file, most of them are self-explanatory but here are the few that are not

  • ticker_begin_block is the block from which you want the epoch_generator service to start (starts from the current block if set to 0)
  • anchor_chain_rpc.url is the RPC URL for Prost Chain where the protocol state lives
  • anchor_chain_rpc.protcol_state_address is the Protocol State contract address with which EpochGenerator interacts and releases Epochs
  • anchor_chain_rpc.validator_epoch_address is the EVM account address for the validator this is releasing/finalizing Epochs
  • anchor_chain_rpc.validator_epoch_private_key is the validator EVM account address private key
  • anchor_chain_rpc.force_consensus_address is the account address for the force consensus service, this doesn't need to be a validator account
  • anchor_chain_rpc.force_consensus_private_key is the private key for the force consensus account

Monitoring and Debugging

Login to the Onchain Consensus Docker container using docker exec -it <container_id> bash (use docker ps to see running containers) and use the following commands for monitoring and debugging:

  • To monitor the status of running processes, run pm2 status.
  • To see all logs, run pm2 logs.
  • To see logs for a specific process, run pm2 logs <Process Identifier>.
  • To see only error logs, run pm2 logs --err.

Or you can simply use docker logs -f onchain-consensus if you don't want to go inside the docker container.

Epoch Generation

An epoch denotes a range of block heights on the data source blockchain, Ethereum mainnet in the case of Uniswap v2. This makes it easier to collect state transitions and snapshots of data on equally spaced block height intervals, as well as to support future work on other lightweight anchor proof mechanisms like Merkle proofs, succinct proofs, etc.

The size of an epoch is configurable. Let that be referred to as size(E).

  • A service keeps track of the head of the chain as it moves ahead, and a marker h₀ against the max block height from the last released epoch. This makes the beginning of the next epoch, h₁ = h₀ + 1.
  • Once the head of the chain has moved sufficiently ahead so that an epoch can be published, an epoch finalization service takes into account the following factors
    • chain reorganization reports where the reorganized limits are a subset of the epoch qualified to be published
    • a configurable ‘offset’ from the bleeding edge of the chain

and then publishes an epoch (h₁, h₂) so that h₂ - h₁ + 1 == size(E). The next epoch, therefore, is tracked from h₂ + 1.

The Epoch Release process is explained in detail in the sequence diagram below Epoch Generator Sequence Diagram

Force Consensus

Force consensus is an optional mechanism that can be run by anyone in the network and is designed to trigger consensus checks for projects that didn't reach consensus automatically with a 51% majority within the submission window. This will force consensus if possible if the project submissions meet all internal criteria for consensus after the submission window is closed.

Force Consensus works slightly differently than Epoch Generator and is heavily optimized to handle a lot of projects. The sequence diagram explaining the flow is given below Force Consensus Sequence Diagram

Transaction tasks are then processed parallelly using the following flow Force Consensus Transaction Task Processing

Running just Consensus service using Docker

If you want to deploy consensus service for some reason, you can do so by following the following steps:

  • Build the image using ./build-docker.sh
  • Run the image using
 docker rm -f onchain-consensus && docker run --add-host host.docker.internal:host-gateway -p 8080:8080 --name onchain-consensus -d powerloom-onchain-consensus:latest && docker logs -f onchain-consensus

This will run the consensus layer on port 9030 of your host.

Consensus Dashboard

The UI dashboard for this is hosted at ap-consensus-dashboard, please follow the deploy instructions there to run the UI.

node-issue-report-collector's People

Contributors

anomit avatar chaitanyaprem avatar getjiggy avatar seth-schmidt avatar swarooph avatar xadahiya avatar

Watchers

avatar avatar avatar avatar avatar avatar

Forkers

thecoderpanda

node-issue-report-collector's Issues

Off chain consensus auth

Moving issue description from legacy audit protocol to present repo

  • UUIDs are a shared secret between snapshotter and consensus services
  • UUIDs to be mapped to some public key (to be used for metrics), name, email..
  • UUIDs cannot be randomly sent - must pre-exist in the redis db

Following describe changes required in Consensus service to achieve the above requirement. In addition to these a CLI would be required which would help provision a new snapshotter which is referred in #212

  • Auth check against snapshotter's uuid to be done for following endpoints from the list of snapshotters stored against snapshotterInfo:allowedSnapshotters:
  • /registerProjectPeer
  • /submitSnapshot
  • /checkForSnapshotConfirmation
  • /epochStatus
  • All the metrics endpoints i.e starting with /metrics that are using snapshotter UUID are to be modified to use/return the alias instead of UUID. The snapshotterInfo:allowedSnapshotters HASH can be used to derive required aliases.

Running snapshotters with lot of projects has causing redis connection limits to be hit

While testing in staging instance with 2 snapshotters with all pair contracts for UNISWAPV2 (which amounts to 372 projects), have noticed some issues in consensus service wrt reaching redis connection limit.

Temporary workaround: Increased redis pool default size in consensus service to 500. This is not scalable as number of snapshotters increase.

The cause for this is that there are multiple redis calls being made per submitSnapshot and checkSnapshotConfirm invocations leading to this issue.

CLI to deactivate snapshotters

We need to have a CLI utility that would help in deactivating a snapshotter in case they are not participating in consensus. CLI can either take uuid/alias as the argument in order to do the same
The CLI in addition to deactivating the snapshotter in the profile, also remove the snapshotter from the projects where registration was done.

Also, review the flows and take care in removing from any other redis data-structure which may affect consensus.

Paginate results of API "/metrics/issues/{time_window}"

https://nms-testnet-reporting.powerloom.io/docs#/

To ensure data manageability and enhance user experience, it is crucial to implement pagination for API responses when dealing with large timestamp ranges. Without pagination, the volume of data retrieved can be substantial, leading to cumbersome navigation and excessive scrolling. Implementing pagination will streamline the presentation of results, making them more accessible and user-friendly.

New Configuration in off-chain-consensus to specify criteria for achieving consensus

Configuration for the criteria to determine consensus for an epoch. Currently it is hard-coded to 2/3rds of registered snapshotters.

Instead have 2 settings for the consensus criteria as given below.

  • The percentage configuration shall be used instead of hard-coded 2/3 to check the consensus status for an epoch as and when a snapshot is submitted by snapshotter and is within submission schedule. The epoch state is already being recorded in redis once consensus is achieved.
  • If consensus is not achieved within the submission schedule (determined by a timer task as per #5) for the epoch, then the configured min_snapshotter_count shall be used to determine if consensus has been achieved for an epoch and mark it as finalized. e.g: If count is set to 2, and submission schedule is passed, then as long as 2 snapshotters submit the same snapshot, the epoch will be considered as finalized and the snapshotCID shall be recorded as finalizedCID.

Note: defaults are given below.

{
    "consensus_criteria":{
        "min_snapshotter_count": 3,
        "percentage": 66
    }
}

Relayer

Either find a third party service or build our own Relayer architecture

More details TBD.

Related to #27

Relayer design and discussion

Image

Top level issue to track Relayer development

Offchain Consensus Modifier (Admin Module)

Offchain Consensus to be modified to handle the following responsibilities

  • Epoch Generation
  • Snapshotter Management
  • Stats and Issues dashboard

Transaction Manager

Ap-transaction-manager to be modified to handle all transactions.
All powerloom agents will submit requests to the relayer (admin module API), which will then forward EIP712 signed transactions to the transaction manager which will submit transactions to chain.

Current State and Todo

Transaction Manager

To be designed

Offchain Consensus Modified (Admin Module)

Things to do -

  • Epoch Generator, Snapshotter Management (Done, might need some modifications once transaction manager is ready)

Scheduled consensus checks on epochs

Current situation

Presently, on every submit_snapshot() there is a calculation done on the set of submitted snapshots and a finalization status is arrived in there.

Why is this a problem

With growing number of snapshotters for a project, this would increase the response time beyond reasonable limits. A consensus calculation logic being triggered for every submission is not even a decentralized, stateful way of doing things.

This also makes it easier to support multiple consensus strategies like fixed N submitters or >67% etc.

Desired solution

For our offchain consensus service, it would suffice to schedule an asyncio.Task or asyncio.Future to be executed at the closing timestamp of an epoch's submission window.

Look into loop.call_later() or loop.call_at() of asyncio

Caveats 🤔

This may have an undesirable side effect for waiting an entire minute or whatever the submission window is to finalize consensus of epochs, which is polled by audit protocols to finalize their copy of the DAG chains, and ultimately affect indexing and aggregation atop these indexes.

Consensus service should return 401 for project APIs if an unregistered uuid is used

For all the non metric API's consensus service should return 401 in case uuid used is not registered with consensus db.

  • Note that this check is required to be added in API's where-ever uuid/instance-ID based auth is done.

As of now 200 is being returned, which is causing clients such as project registration to consider request as success.

Switch to custom Gunicorn implementation for better performance

Describe the bug

Current consensus startup uses command line Gunicorn setup which is causing performance issues because it does not support Resource RLimit configuration

Expected behavior
Application workers should be started through a custom Gunicorn implementation will provide better control over Resource limits and other process config.

Proposed Solution
Implement a custom Gunicorn Application class to start workers instead.

Consensus service throws 500 for /metrics/projects

Issue

When trying to access consensus service sometimes http://localhost:9030/metrics/projects it throws internal server error.
Below is the log trace for the same.

Reason

By default uuid is set to None (if not provided) and redis throws error when trying to save the object for some reason (see error trace below)

Logtrace

January 16, 2023 > 10:18:19 | ERROR | Request failed: Invalid input of type: 'NoneType'. Convert to a bytes, string, int or float first.| {'request_id': '88f0e895-65e7-491a-80d6-31a9efe4ce2b', 'service': 'PowerLoom|OffChainConsensus|ServiceEntry'}
Traceback (most recent call last):

  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/anyio/streams/memory.py", line 94, in receive
    return self.receive_nowait()
           │    └ <function MemoryObjectReceiveStream.receive_nowait at 0x104f273a0>
           └ MemoryObjectReceiveStream(_state=MemoryObjectStreamState(max_buffer_size=0, buffer=deque([]), open_send_channels=0, open_rece...
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/anyio/streams/memory.py", line 89, in receive_nowait
    raise WouldBlock
          └ <class 'anyio.WouldBlock'>

anyio.WouldBlock


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/middleware/base.py", line 77, in call_next
    message = await recv_stream.receive()
                    │           └ <function MemoryObjectReceiveStream.receive at 0x104f27430>
                    └ MemoryObjectReceiveStream(_state=MemoryObjectStreamState(max_buffer_size=0, buffer=deque([]), open_send_channels=0, open_rece...
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/anyio/streams/memory.py", line 114, in receive
    raise EndOfStream
          └ <class 'anyio.EndOfStream'>

anyio.EndOfStream


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/Users/xadahiya/miniconda3/envs/powerloom/bin/gunicorn", line 8, in <module>
    sys.exit(run())
    │   │    └ <function run at 0x1030c9a60>
    │   └ <built-in function exit>
    └ <module 'sys' (built-in)>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
    └ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/gunicorn/app/base.py", line 231, in run
    super().run()
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/gunicorn/app/base.py", line 72, in run
    Arbiter(self).run()
    │       └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x1030cc130>
    └ <class 'gunicorn.arbiter.Arbiter'>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/gunicorn/arbiter.py", line 202, in run
    self.manage_workers()
    │    └ <function Arbiter.manage_workers at 0x103afa700>
    └ <gunicorn.arbiter.Arbiter object at 0x1030cc100>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
    self.spawn_workers()
    │    └ <function Arbiter.spawn_workers at 0x103afa820>
    └ <gunicorn.arbiter.Arbiter object at 0x1030cc100>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
    self.spawn_worker()
    │    └ <function Arbiter.spawn_worker at 0x103afa790>
    └ <gunicorn.arbiter.Arbiter object at 0x1030cc100>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
    worker.init_process()
    │      └ <function UvicornWorker.init_process at 0x1042b6a60>
    └ <uvicorn.workers.UvicornWorker object at 0x1042c42e0>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/uvicorn/workers.py", line 66, in init_process
    super(UvicornWorker, self).init_process()
          │              └ <uvicorn.workers.UvicornWorker object at 0x1042c42e0>
          └ <class 'uvicorn.workers.UvicornWorker'>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/gunicorn/workers/base.py", line 142, in init_process
    self.run()
    │    └ <function UvicornWorker.run at 0x1042b6ca0>
    └ <uvicorn.workers.UvicornWorker object at 0x1042c42e0>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/uvicorn/workers.py", line 98, in run
    return asyncio.run(self._serve())
           │       │   │    └ <function UvicornWorker._serve at 0x1042b6c10>
           │       │   └ <uvicorn.workers.UvicornWorker object at 0x1042c42e0>
           │       └ <function run at 0x103cd2700>
           └ <module 'asyncio' from '/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/asyncio/__init__.py'>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
           │    │                  └ <coroutine object UvicornWorker._serve at 0x105405ec0>
           │    └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
           └ <uvloop.Loop running=True closed=False debug=False>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 419, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
                   └ <uvicorn.middleware.proxy_headers.ProxyHeadersMiddleware object at 0x105482b20>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
                 │    │   │      │        └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x10558f070>>
                 │    │   │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x10558f0...
                 │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
                 │    └ <fastapi.applications.FastAPI object at 0x10541b910>
                 └ <uvicorn.middleware.proxy_headers.ProxyHeadersMiddleware object at 0x105482b20>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/fastapi/applications.py", line 270, in __call__
    await super().__call__(scope, receive, send)
                           │      │        └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x10558f070>>
                           │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x10558f0...
                           └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
          │    │                │      │        └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x10558f070>>
          │    │                │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x10558f0...
          │    │                └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
          │    └ <starlette.middleware.errors.ServerErrorMiddleware object at 0x105429310>
          └ <fastapi.applications.FastAPI object at 0x10541b910>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
          │    │   │      │        └ <function ServerErrorMiddleware.__call__.<locals>._send at 0x105576160>
          │    │   │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x10558f0...
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
          │    └ <starlette.middleware.base.BaseHTTPMiddleware object at 0x1054292b0>
          └ <starlette.middleware.errors.ServerErrorMiddleware object at 0x105429310>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/middleware/base.py", line 106, in __call__
    response = await self.dispatch_func(request, call_next)
                     │    │             │        └ <function BaseHTTPMiddleware.__call__.<locals>.call_next at 0x1055760d0>
                     │    │             └ <starlette.requests.Request object at 0x10558f850>
                     │    └ <function request_middleware at 0x10541dc10>
                     └ <starlette.middleware.base.BaseHTTPMiddleware object at 0x1054292b0>

> File "/Users/xadahiya/workspace/offchain-consensus/consensus_entry_point.py", line 81, in request_middleware
    response = await call_next(request)
                     │         └ <starlette.requests.Request object at 0x10558f850>
                     └ <function BaseHTTPMiddleware.__call__.<locals>.call_next at 0x1055760d0>

  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/middleware/base.py", line 80, in call_next
    raise app_exc
          └ DataError("Invalid input of type: 'NoneType'. Convert to a bytes, string, int or float first.")
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/middleware/base.py", line 69, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
          │    │   │      │                      └ <function BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.send_no_error at 0x105576dc0>
          │    │   │      └ <function BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.receive_or_disconnect at 0x105576af0>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
          │    └ <starlette.middleware.cors.CORSMiddleware object at 0x105429250>
          └ <starlette.middleware.base.BaseHTTPMiddleware object at 0x1054292b0>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
          │    │   │      │        └ <function BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.send_no_error at 0x105576dc0>
          │    │   │      └ <function BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.receive_or_disconnect at 0x105576af0>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
          │    └ <starlette.middleware.exceptions.ExceptionMiddleware object at 0x105429220>
          └ <starlette.middleware.cors.CORSMiddleware object at 0x105429250>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
          │    │   │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x1055d45e0>
          │    │   │      └ <function BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.receive_or_disconnect at 0x105576af0>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
          │    └ <fastapi.middleware.asyncexitstack.AsyncExitStackMiddleware object at 0x1054291c0>
          └ <starlette.middleware.exceptions.ExceptionMiddleware object at 0x105429220>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
          │    │   │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x1055d45e0>
          │    │   │      └ <function BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.receive_or_disconnect at 0x105576af0>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
          │    └ <fastapi.routing.APIRouter object at 0x10541b970>
          └ <fastapi.middleware.asyncexitstack.AsyncExitStackMiddleware object at 0x1054291c0>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
          │     │      │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x1055d45e0>
          │     │      │      └ <function BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.receive_or_disconnect at 0x105576af0>
          │     │      └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
          │     └ <function Route.handle at 0x10506da60>
          └ APIRoute(path='/metrics/projects', name='get_projects', methods=['GET'])
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
          │    │   │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x1055d45e0>
          │    │   │      └ <function BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.receive_or_disconnect at 0x105576af0>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
          │    └ <function request_response.<locals>.app at 0x105431940>
          └ APIRoute(path='/metrics/projects', name='get_projects', methods=['GET'])
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
                     │    └ <starlette.requests.Request object at 0x1055c1220>
                     └ <function get_request_handler.<locals>.app at 0x1054318b0>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/fastapi/routing.py", line 225, in app
    solved_result = await solve_dependencies(
                          └ <function solve_dependencies at 0x105054d30>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/fastapi/dependencies/utils.py", line 533, in solve_dependencies
    solved = await call(**sub_values)
                   │      └ {'auth_check_dep': AuthCheck(authorized=True, api_key='public', reason='', owner=SnapshotterMetadata(rate_limit='10000/day;20...
                   └ <function rate_limit_auth_check at 0x1053c7310>

  File "/Users/xadahiya/workspace/offchain-consensus/auth/helpers/helpers.py", line 190, in rate_limit_auth_check
    await auth_redis_conn.hset(
          │               └ <function HashCommands.hset at 0x10517d700>
          └ Redis<ConnectionPool<Connection<host=127.0.0.1,port=6371,db=1>>>

  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/redis/asyncio/client.py", line 505, in execute_command
    return await conn.retry.call_with_retry(
                 │    │     └ <function Retry.call_with_retry at 0x105274f70>
                 │    └ <redis.asyncio.retry.Retry object at 0x105479a80>
                 └ Connection<host=127.0.0.1,port=6371,db=1>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/redis/asyncio/retry.py", line 59, in call_with_retry
    return await do()
                 └ <function Redis.execute_command.<locals>.<lambda> at 0x1055d4670>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/redis/asyncio/client.py", line 480, in _send_command_parse_response
    await conn.send_command(*args)
          │    │             └ ('HSET', 'user:127.0.0.1', 'rate_limit', '10000/day;20/minute;2/second', 'active', <UserStatusEnum.active: 'active'>, 'callsC...
          │    └ <function Connection.send_command at 0x1052859d0>
          └ Connection<host=127.0.0.1,port=6371,db=1>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/redis/asyncio/connection.py", line 806, in send_command
    self.pack_command(*args), check_health=kwargs.get("check_health", True)
    │    │             │                   │      └ <method 'get' of 'dict' objects>
    │    │             │                   └ {}
    │    │             └ ('HSET', 'user:127.0.0.1', 'rate_limit', '10000/day;20/minute;2/second', 'active', <UserStatusEnum.active: 'active'>, 'callsC...
    │    └ <function Connection.pack_command at 0x105285b80>
    └ Connection<host=127.0.0.1,port=6371,db=1>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/redis/asyncio/connection.py", line 881, in pack_command
    for arg in map(self.encoder.encode, args):
        │          │    │               └ (b'HSET', 'user:127.0.0.1', 'rate_limit', '10000/day;20/minute;2/second', 'active', <UserStatusEnum.active: 'active'>, 'calls...
        │          │    └ <member 'encoder' of 'Connection' objects>
        │          └ Connection<host=127.0.0.1,port=6371,db=1>
        └ b'uuid'
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/redis/asyncio/connection.py", line 123, in encode
    raise DataError(
          └ <class 'redis.exceptions.DataError'>

redis.exceptions.DataError: Invalid input of type: 'NoneType'. Convert to a bytes, string, int or float first.
January 16, 2023 > 10:18:19 | ERROR | Request failed: Invalid input of type: 'NoneType'. Convert to a bytes, string, int or float first.| {'request_id': '88f0e895-65e7-491a-80d6-31a9efe4ce2b', 'service': 'PowerLoom|OffChainConsensus|ServiceEntry'}
Traceback (most recent call last):

  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/anyio/streams/memory.py", line 94, in receive
    return self.receive_nowait()
           │    └ <function MemoryObjectReceiveStream.receive_nowait at 0x104f273a0>
           └ MemoryObjectReceiveStream(_state=MemoryObjectStreamState(max_buffer_size=0, buffer=deque([]), open_send_channels=0, open_rece...
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/anyio/streams/memory.py", line 89, in receive_nowait
    raise WouldBlock
          └ <class 'anyio.WouldBlock'>

anyio.WouldBlock


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/middleware/base.py", line 77, in call_next
    message = await recv_stream.receive()
                    │           └ <function MemoryObjectReceiveStream.receive at 0x104f27430>
                    └ MemoryObjectReceiveStream(_state=MemoryObjectStreamState(max_buffer_size=0, buffer=deque([]), open_send_channels=0, open_rece...
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/anyio/streams/memory.py", line 114, in receive
    raise EndOfStream
          └ <class 'anyio.EndOfStream'>

anyio.EndOfStream


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/Users/xadahiya/miniconda3/envs/powerloom/bin/gunicorn", line 8, in <module>
    sys.exit(run())
    │   │    └ <function run at 0x1030c9a60>
    │   └ <built-in function exit>
    └ <module 'sys' (built-in)>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
    └ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/gunicorn/app/base.py", line 231, in run
    super().run()
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/gunicorn/app/base.py", line 72, in run
    Arbiter(self).run()
    │       └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x1030cc130>
    └ <class 'gunicorn.arbiter.Arbiter'>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/gunicorn/arbiter.py", line 202, in run
    self.manage_workers()
    │    └ <function Arbiter.manage_workers at 0x103afa700>
    └ <gunicorn.arbiter.Arbiter object at 0x1030cc100>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
    self.spawn_workers()
    │    └ <function Arbiter.spawn_workers at 0x103afa820>
    └ <gunicorn.arbiter.Arbiter object at 0x1030cc100>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
    self.spawn_worker()
    │    └ <function Arbiter.spawn_worker at 0x103afa790>
    └ <gunicorn.arbiter.Arbiter object at 0x1030cc100>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
    worker.init_process()
    │      └ <function UvicornWorker.init_process at 0x1042b6a60>
    └ <uvicorn.workers.UvicornWorker object at 0x1042c42e0>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/uvicorn/workers.py", line 66, in init_process
    super(UvicornWorker, self).init_process()
          │              └ <uvicorn.workers.UvicornWorker object at 0x1042c42e0>
          └ <class 'uvicorn.workers.UvicornWorker'>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/gunicorn/workers/base.py", line 142, in init_process
    self.run()
    │    └ <function UvicornWorker.run at 0x1042b6ca0>
    └ <uvicorn.workers.UvicornWorker object at 0x1042c42e0>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/uvicorn/workers.py", line 98, in run
    return asyncio.run(self._serve())
           │       │   │    └ <function UvicornWorker._serve at 0x1042b6c10>
           │       │   └ <uvicorn.workers.UvicornWorker object at 0x1042c42e0>
           │       └ <function run at 0x103cd2700>
           └ <module 'asyncio' from '/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/asyncio/__init__.py'>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
           │    │                  └ <coroutine object UvicornWorker._serve at 0x105405ec0>
           │    └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
           └ <uvloop.Loop running=True closed=False debug=False>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 419, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
                   └ <uvicorn.middleware.proxy_headers.ProxyHeadersMiddleware object at 0x105482b20>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
                 │    │   │      │        └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x10558f070>>
                 │    │   │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x10558f0...
                 │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
                 │    └ <fastapi.applications.FastAPI object at 0x10541b910>
                 └ <uvicorn.middleware.proxy_headers.ProxyHeadersMiddleware object at 0x105482b20>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/fastapi/applications.py", line 270, in __call__
    await super().__call__(scope, receive, send)
                           │      │        └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x10558f070>>
                           │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x10558f0...
                           └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
          │    │                │      │        └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x10558f070>>
          │    │                │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x10558f0...
          │    │                └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
          │    └ <starlette.middleware.errors.ServerErrorMiddleware object at 0x105429310>
          └ <fastapi.applications.FastAPI object at 0x10541b910>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
          │    │   │      │        └ <function ServerErrorMiddleware.__call__.<locals>._send at 0x105576160>
          │    │   │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x10558f0...
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
          │    └ <starlette.middleware.base.BaseHTTPMiddleware object at 0x1054292b0>
          └ <starlette.middleware.errors.ServerErrorMiddleware object at 0x105429310>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/middleware/base.py", line 106, in __call__
    response = await self.dispatch_func(request, call_next)
                     │    │             │        └ <function BaseHTTPMiddleware.__call__.<locals>.call_next at 0x1055760d0>
                     │    │             └ <starlette.requests.Request object at 0x10558f850>
                     │    └ <function request_middleware at 0x10541dc10>
                     └ <starlette.middleware.base.BaseHTTPMiddleware object at 0x1054292b0>

> File "/Users/xadahiya/workspace/offchain-consensus/consensus_entry_point.py", line 81, in request_middleware
    response = await call_next(request)
                     │         └ <starlette.requests.Request object at 0x10558f850>
                     └ <function BaseHTTPMiddleware.__call__.<locals>.call_next at 0x1055760d0>

  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/middleware/base.py", line 80, in call_next
    raise app_exc
          └ DataError("Invalid input of type: 'NoneType'. Convert to a bytes, string, int or float first.")
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/middleware/base.py", line 69, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
          │    │   │      │                      └ <function BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.send_no_error at 0x105576dc0>
          │    │   │      └ <function BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.receive_or_disconnect at 0x105576af0>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
          │    └ <starlette.middleware.cors.CORSMiddleware object at 0x105429250>
          └ <starlette.middleware.base.BaseHTTPMiddleware object at 0x1054292b0>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
          │    │   │      │        └ <function BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.send_no_error at 0x105576dc0>
          │    │   │      └ <function BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.receive_or_disconnect at 0x105576af0>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
          │    └ <starlette.middleware.exceptions.ExceptionMiddleware object at 0x105429220>
          └ <starlette.middleware.cors.CORSMiddleware object at 0x105429250>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
          │    │   │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x1055d45e0>
          │    │   │      └ <function BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.receive_or_disconnect at 0x105576af0>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
          │    └ <fastapi.middleware.asyncexitstack.AsyncExitStackMiddleware object at 0x1054291c0>
          └ <starlette.middleware.exceptions.ExceptionMiddleware object at 0x105429220>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
          │    │   │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x1055d45e0>
          │    │   │      └ <function BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.receive_or_disconnect at 0x105576af0>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
          │    └ <fastapi.routing.APIRouter object at 0x10541b970>
          └ <fastapi.middleware.asyncexitstack.AsyncExitStackMiddleware object at 0x1054291c0>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
          │     │      │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x1055d45e0>
          │     │      │      └ <function BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.receive_or_disconnect at 0x105576af0>
          │     │      └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
          │     └ <function Route.handle at 0x10506da60>
          └ APIRoute(path='/metrics/projects', name='get_projects', methods=['GET'])
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
          │    │   │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x1055d45e0>
          │    │   │      └ <function BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.receive_or_disconnect at 0x105576af0>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 9031), 'cl...
          │    └ <function request_response.<locals>.app at 0x105431940>
          └ APIRoute(path='/metrics/projects', name='get_projects', methods=['GET'])
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
                     │    └ <starlette.requests.Request object at 0x1055c1220>
                     └ <function get_request_handler.<locals>.app at 0x1054318b0>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/fastapi/routing.py", line 225, in app
    solved_result = await solve_dependencies(
                          └ <function solve_dependencies at 0x105054d30>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/fastapi/dependencies/utils.py", line 533, in solve_dependencies
    solved = await call(**sub_values)
                   │      └ {'auth_check_dep': AuthCheck(authorized=True, api_key='public', reason='', owner=SnapshotterMetadata(rate_limit='10000/day;20...
                   └ <function rate_limit_auth_check at 0x1053c7310>

  File "/Users/xadahiya/workspace/offchain-consensus/auth/helpers/helpers.py", line 190, in rate_limit_auth_check
    await auth_redis_conn.hset(
          │               └ <function HashCommands.hset at 0x10517d700>
          └ Redis<ConnectionPool<Connection<host=127.0.0.1,port=6371,db=1>>>

  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/redis/asyncio/client.py", line 505, in execute_command
    return await conn.retry.call_with_retry(
                 │    │     └ <function Retry.call_with_retry at 0x105274f70>
                 │    └ <redis.asyncio.retry.Retry object at 0x105479a80>
                 └ Connection<host=127.0.0.1,port=6371,db=1>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/redis/asyncio/retry.py", line 59, in call_with_retry
    return await do()
                 └ <function Redis.execute_command.<locals>.<lambda> at 0x1055d4670>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/redis/asyncio/client.py", line 480, in _send_command_parse_response
    await conn.send_command(*args)
          │    │             └ ('HSET', 'user:127.0.0.1', 'rate_limit', '10000/day;20/minute;2/second', 'active', <UserStatusEnum.active: 'active'>, 'callsC...
          │    └ <function Connection.send_command at 0x1052859d0>
          └ Connection<host=127.0.0.1,port=6371,db=1>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/redis/asyncio/connection.py", line 806, in send_command
    self.pack_command(*args), check_health=kwargs.get("check_health", True)
    │    │             │                   │      └ <method 'get' of 'dict' objects>
    │    │             │                   └ {}
    │    │             └ ('HSET', 'user:127.0.0.1', 'rate_limit', '10000/day;20/minute;2/second', 'active', <UserStatusEnum.active: 'active'>, 'callsC...
    │    └ <function Connection.pack_command at 0x105285b80>
    └ Connection<host=127.0.0.1,port=6371,db=1>
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/redis/asyncio/connection.py", line 881, in pack_command
    for arg in map(self.encoder.encode, args):
        │          │    │               └ (b'HSET', 'user:127.0.0.1', 'rate_limit', '10000/day;20/minute;2/second', 'active', <UserStatusEnum.active: 'active'>, 'calls...
        │          │    └ <member 'encoder' of 'Connection' objects>
        │          └ Connection<host=127.0.0.1,port=6371,db=1>
        └ b'uuid'
  File "/Users/xadahiya/miniconda3/envs/powerloom/lib/python3.9/site-packages/redis/asyncio/connection.py", line 123, in encode
    raise DataError(
          └ <class 'redis.exceptions.DataError'>

redis.exceptions.DataError: Invalid input of type: 'NoneType'. Convert to a bytes, string, int or float first

Enhancing the forceConsensus Mechanism for Greater Flexibility and Compatibility

Problem Description:
Currently, the forceConsensus mechanism is integrated within the Epoch Generator service, which poses challenges in supporting chains with smaller block times, such as Polygon. Moreover, the dependency of forceConsensus on the epoch release window makes it less flexible and less effective, especially when dealing with submissionWindow sizes larger than the epoch release time. This limitation is likely to occur in several chains with smaller sizes.

Proposed Solution:
To address these issues, it is essential to refactor and redesign the forceConsensus mechanism into an independent service that not only improves performance but also ensures greater flexibility and compatibility across different blockchain networks.

  1. Separation of Epoch Generator and Force Consensus:
    The first step is to completely separate the Epoch Generator and Force Consensus services. By doing so, we can enable better modularity and allow each service to focus on its specific functionality without unnecessary dependencies.

  2. Redesigned Force Consensus Service:
    The redesign of the Force Consensus service should emphasize speed, flexibility, and generic support for varying submissionWindow sizes, regardless of the epoch release times. This will be achieved through the implementation of efficient algorithms and optimized data structures.

  3. Compatibility with Chains of Different Sizes:
    The refactored Force Consensus service should be capable of seamlessly integrating with chains of varying sizes. Special attention will be given to supporting chains with smaller block times like Polygon, making the solution more versatile and adaptable.

Support bulk queries of epochs in `/epochStatus`

Is your feature request related to a problem?

Offchain consensus service opens up an API endpoint, /epochStatus, which is used by the DAG finalizer component in Audit Protocol to perform self-healing of the DAG chain of snapshots maintained by each individual snapshotter. This supports only one epoch to be queried at a time.

https://github.com/PowerLoom/offchain-consensus/blob/c2aff8827941c02c1bc15852e8e5fea258fbd465/consensus_entry_point.py#L300-L315

At the moment, if there are multitude of chains along with a large span of blocks corresponding to the local state of a snapshotter that have fallen behind, the self-healing feature would cause a flood of requests to the above endpoint thereby hitting rate limits imposed on snapshotter API requests and finally causing the self healing to fail.

Describe the solution you'd like

Support an array of epochs to be passed in the above request. Like the following:

@app.post('/epochStatus')
async def epoch_status(
        request: Request,
        req_parsed: EpochStatusRequest,
        response: Response,
        rate_limit_auth_dep: RateLimitAuthCheck = Depends(rate_limit_auth_check)
)

class EpochStatusRequest(PeerUUIDIncludedRequests):
    projectID: str
    epochs: List[EpochBase]

Describe alternatives you've considered
Not applicable.

Additional context

Project rename and overhaul

Is your feature request related to a problem?
NA

Describe the solution you'd like
Currently all other services included in the main branch have been moved out to their own repositories, except the server that collects issue reports from nodes. The project name should be changed appropriately as well as the code for the other services removed from this repository.

Describe alternatives you've considered
NA

Additional context
NA

Fix Epoch Generator to not use env variable on subsequent restarts

Issue

If we start epoch-tracker for the first time by setting TICKER_BEGIN_BLOCK environment variable, on every subsequent restart the same value of TICKER_BEGIN_BLOCK is picked up causing it to fail with below error log.

January 14, 2023 > 06:22:10 | DEBUG | Attempting to start from 16399031 but last block 16403210 found in Redis.| {'module': 'PowerLoom|OffChainConsensus|EpochGenerator'}
January 14, 2023 > 06:22:10 | DEBUG | Either use clean_slate_ticker.py to reset the Redis state or remove the begin_block_epoch argument.| {'module': 'PowerLoom|OffChainConsensus|EpochGenerator'}
January 14, 2023 > 06:22:10 | DEBUG | Shutting down| {'module': 'PowerLoom|OffChainConsensus|EpochGenerator'}
Set sim mode:  False
January 14, 2023 > 06:22:11 | DEBUG | Attempting to start from 16399031 but last block 16403210 found in Redis.| {'module': 'PowerLoom|OffChainConsensus|EpochGenerator'}
January 14, 2023 > 06:22:11 | DEBUG | Either use clean_slate_ticker.py to reset the Redis state or remove the begin_block_epoch argument.| {'module': 'PowerLoom|OffChainConsensus|EpochGenerator'}
January 14, 2023 > 06:22:11 | DEBUG | Shutting down| {'module': 'PowerLoom|OffChainConsensus|EpochGenerator'}
Set sim mode:  False

The env variable should only be used for starting the first time or after clean-slate, and subsequently the process should use lastGeneratedEpoch stored in redis.

The root-cause seems to be as pm2 is caching this env variable and using it for every restart.

Work-Around

Currently the work-around is to delete the existing pm2 epoch-tracker process start it freshly by not setting the env variable.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.