synthetixio / futures-keepers Goto Github PK

It looks green but is failing? jq undefined.

Looking into increasing the observability of the keeper in production:

• alerts for keeper wallet balances (sUSD, ETH)

• list of kept markets

• list of open positions

• list of positions at risk of liquidation

• alerts for keeper crashes/restarts

• graph of keeper ETH/sUSD balance over time

• graphs of # orders liquidated over time for each market

• Setup devops infra with AWS cloudwatch. (see https://github.com/Synthetixio/snapshot-keeper).

• Setup alerts for keeper wallet balances.

• Add mechanism to withdraw liquidation rewards from accounts.

Convert

        console.log(
          "FuturesMarket",
          this.baseAsset,
          blue("PositionModified"),
          `[id=${id} account=${account}]`
        );

Failures:

• ETH RPC node disconnects.
• Unexpected error (eg. nonce error).

Metrics

• eth node connection drops
• new blocks on ovm testnet
• keeper restarts

Latency 
    How long it takes to liquidate a position.
Traffic
    How many new blocks are coming through?
    How many new positions are being opened/closed?
Errors
    ETH RPC node Failures
    Keeper wallet out of ETH
    Application crash due to code error
Saturation

Store and collect logs.