hifi-finance / hifi-liquidator Goto Github PK

View Code? Open in Web Editor NEW

11.0 5.0 2.0 144 KB

Utility for automatically liquidating underwater accounts in Hifi

Rust 98.98% Dockerfile 1.02%

ethereum defi decentralized-finance rust

hifi-liquidator's Introduction

Hifi Liquidator

Utility for automatically liquidating underwater accounts in Hifi.

Limitations

Compatible only with the WBTC/USDC pair

hifi-liquidator's People

Contributors

Stargazers

Watchers

Forkers

lucky-zcy ccamateur

hifi-liquidator's Issues

Save vault data in a database

We're currently dumping the vault data into a JSON file on disk:

https://github.com/MainframeHQ/hifi-liquidator/blob/007c2d3255c5067788da93dde5c915820b001a3c/liquidator/src/sentinel.rs#L150-L159

There many downsides to it, but the primary issue is that the data gets pruned every time a Docker container is pruned. We should switch to a database model and use Docker Compose to connect to the database container.

Ensure that clutchable collateral is sufficient

Calling liquidateBorrow on the fyToken reverts when the price of the collateral has fallen so much that the liquidator cannot receive their 10% share of the deal (an economic risk that we were well aware of when we wrote the protocol).

Before liquidating the user via the Uniswap V2 pair, we should call the getClutchableCollateral function on the BalanceSheet and compare it to the currently locked collateral in the vault.

Only replace the pending transaction if the gas has been bumped

It is not a given that the gas will always be bumped by the escalator. The logic in GeometricGasPrice (part of ethers-rs) bumps the gas only if more than every_secs seconds passed. Therefore, it is possible that a transaction will be replaced with the same gas price, which causes the tx to revert (at least in Parity).

The solution is to modify the remove_pending_tx_tuple_or_bump_gas_price_internal function to account for this edge case.

Handle revert reasons on Kovan

Refer to Do all Ethereum networks return the revert reasons as a “message” field? for a thorough explanation of the issue.

We should modify the Err variant in liquidations.rs so that if the network used is Kovan, the revert reason is decoded from the data field.

Restrict monitoring to vaults with at least a minimum amount of collateral

Monitoring vaults with less than ~$200 worth of collateral may not be worth it, due to gas fees.

Handle node API errors

Whenever Infura returns the following error (and presumably others like it):

Error: Deserialization Error: expected value at line 1 column 1. Response:
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
</body>
</html>

The liquidator program exits, which is not good.

Handle errors returned by the call to UniswapV2Pair

We should replace the code on line 110 in liquidator.rs:

https://github.com/MainframeHQ/hifi-liquidator/blob/c4c4483dc6055a46df2ff7e1ab367d05f97aad6d/liquidator/src/liquidator.rs#L110

With a match expression that checks if the enum variant is Err, and log the error accordingly.

Handle transaction reverts more gracefully

Description

As of d3b1c4d the least, the bot behaves like this:

When it finds an under-collateralized borrower, it triggers a liquidation tx
If the tx hasn't been processed quickly, the bot replaces the tx with one with a higher gas price
If either the original tx or the replacement tx reverts (insufficient minimum profit, insufficient liquidity in Uniswap, insufficient locked collateral when the price falls hard, etc.), the bot enters a never-ending loop by repeating step #1

It's not clear what's the appropriate way to handle this issue. On the one hand, not repeating step #1 would be bad from a security POV - under-collateralized borrowers should be liquidated ASAP. On the other hand, sending a bajillion txs hard-stresses our RPC endpoint, increasing our bill.

Possible Solution

Be discriminate with error handling (in the handle_reverted_tx):

"ERR_INSUFFICIENT_PROFIT": repeating the cycle is fine, since the liquidation could become profitable after the Uniswap pool updates its reserves
"UniswapV2: INSUFFICIENT_LIQUIDITY": same as before
"ERR_INSUFFICIENT_LOCKED_COLLATERAL": wait for the price to bounce back up before repeating the cycle
Generic revert: panic the Rust program to investigate what happened, but only if this wasn't caused by the replacement of a tx

Sometimes the gas is not calculated correctly

See:

Avoid cloning the client on every block

The JSON-RPC client gets cloned on every new block, in the call to update the vaults in the sentinel.rs file. This is putting a heavy load on our EC2 instance - let's figure out a smarter way to structure the data so that we clone the vault less often.

Throw error on WS connection errors

When the WS connection to Infura is closed, the bot should exit the process, so that Kubernetes picks it up and restarts the pod.

Fix the occasional bogus json written to the database file

Sometimes, the data that gets written to the database file ends up being this bogus json:

{
  "last_block": "0x15fdfa6",
  "vaults": {
    "0xae4480e53a2d2ec060732a354d86becae890781f": {
      "0xe844453eb945cea7326ac9e7b1f85f2903f4df2f": {
        "debt": "0x0",
        "debt_in_underlying": "0x0",
        "is_account_underwater": false,
        "locked_collateral": "0x2496"
      },
      "0x22e7e1a5f96b09cdf786fb18e0dba4bc9b017343": {
        "debt": "0x0",
        "debt_in_underlying": "0x0",
        "is_account_underwater": false,
        "locked_collateral": "0x1fc1e20"
      },
      "0x4bcb303609f19e71ab82a3a3393c46bfea1e44fc": {
        "debt": "0x0",
        "debt_in_underlying": "0x0",
        "is_account_underwater": false,
        "locked_collateral": "0x5f5e100"
      },
      "0x9c41b359c12d924c3ff232583c6a1ff04b8aed18": {
        "debt": "0x362907144269dfda56",
        "debt_in_underlying": "0x3b8cc255",
        "is_account_underwater": false,
        "locked_collateral": "0x1565aab700"
      }
    }
  }
}
ollateral":"0x5f5e100"}}}}

Notice the extraneous ollateral":"0x5f5e100"}}}} at the end.

Racing condition when replacing txs

There is a bug with the tx replacement logic.

Sentinel triggers liquidation 0x123
Sentinel sees that 0x123 wasn't mined quickly enough, so it broadcasts a replacement tx 0xabc with a higher gas price.
At the same time with ending step #2, tx 0x123 gets mined.
0xabc will revert with the reason "ERR_ACCOUNT_NOT_UNDERWATER"
The "if let" expression on line 208 in liquidations.rs won't return true, because the 0xabc has been dropped and (I assume) Ethereum gives no receipt for these txs.
The sentinel will keep attempting to replace 0xabc forever, even if 0x123 was mined, but it will fail to replace it. The Err variant will be used when broadcasting the replacement of the replacement tx.

Make the implementation more general-purpose

The current implementation (as per commit 7b5f6f3 at least) is limited to the WBTC/USDC pair. We should do the following to make the implementation more general-purpose:

Query the underlying precision scalar from the FyToken contract instead of using the hardcoded value in vaults.rs