Issue with liquidations about liquidator-v3 HOT 5 CLOSED

@riordanp Yup, removing the ENDPOINT_URL seems to have fixed the problem. For some reason, I thought that the default RPC was the Serum one.

I'll look into some private RPC solutions. They seem to be quite expensive but, I'm curious to try it out. Maybe I can deploy both the Liquidator bot + RPC on Google Cloud and see how that goes.

Thank you again for the help and the awesome bot! :)

from liquidator-v3.

UPDATE: I was running the bot during the huge flash crash yesterday and the bot managed to pick up a few liquidations! It was super cool to see it working but it seems way harder to get a successful liquidation during normal/less volatile periods.

I'm guessing this has to do with the fact that there are other people doing the same thing with more advanced setups and private RPCs? Curious to see how the bot can be improved so that it is competitive/functional in all market conditions besides huge liquidation events.

I'm also still trying to understand the code and logic but during the time I was running the bot, I also noticed another piece of weird behavior I still can't understand.

For the past 10-12 hours the bot has been spamming out transactions constantly for the same two sick accounts which are not in a liquidatable state. It goes through a loop of sending out a transaction, having it fail since the accounts are not liquidatable, and then retrying the same thing. Perhaps the two accounts are glitched on Mango's end and it's messing with the bot?

Screenshots below for the two accounts below:

Thank you again!

from liquidator-v3.

Hi @krazzarkov

The most common error I've encountered is Transaction failed: MangoErrorCode::InvalidParam

You're correct, this one is someone getting there before you when attempting to trigger an advanced order. Will be removing this error message in a future version as it's not particularly helpful.

Besides this error, I have experienced the following timeout issue. This error usually results after the CLI spams sending tx for a few minutes.

Failed to liquidate account 3cj8aQtAFe2rzXhYVP2wUBZJVutSeKJDCh3bkjwJukce Error: "BlockhashNotFound"

These are both due to RPC node problems. This is quite common especially with public RPC services.

Watching accounts... Liquidated account 7nx81DsbC7rmANwZbykVxR3TsQ1HXBhVhAaGYi5TKt41

Not totally sure on this one, adding some new logging soon that should make it clearer what happens in this case.

Failed to liquidate account 8zq5hsJgtCDvMe432N1gzjmyetzrutse8XwjgeWCyFCh Error: Transaction failed: MangoErrorCode::InvalidPriceCache Cache the oracle price to resolve; src/state.rs:588

This is a wider issue with Mango at the moment especially in periods of high load. You would have liquidated but the price cache was stale.

I have also tried playing around and modifying INTERVAL, INTERVAL_ACCOUNTS, and INTERVAL_WEBSOCKET so they are faster however

Changing INTERVAL_ACCOUNTS and INTERVAL_WEBSOCKET will not really make any difference to you liquidation success nor will having a higher account value.. They just control how long to wait between reloading the account list (most account updates come in on websocket anyway, this is just a failsafe), and how long to wait before reconnecting to the websocket. The loop that reloads prices and checks accounts health will run at the same speed regardless.

I was running the bot during the huge flash crash yesterday and the bot managed to pick up a few liquidations! It was super cool to see it working but it seems way harder to get a successful liquidation during normal/less volatile periods.

Glad you got some liquidations! Most of the time when the market is more stable there are very few liquidations happening at all. During periods of volatility there is a large amount. This has been my experience too. There are no doubt people out there running private RPCs with very low latency to the liquidator, and probably some with their own liquidator implementations or modifications to make it faster.

For the past 10-12 hours the bot has been spamming out transactions constantly for the same two sick accounts which are not in a liquidatable state.

Might be caused by RPC caching. Which endpoint do you use? Can you try a different one?

from liquidator-v3.

Hi @riordanp,

Thank you for getting back to me and answering all my questions!

For the error messages, all your explanations made sense to me.

Regarding my last issue with the same two accounts trying to be liquidated by the bot, it definitely seems to be an RPC caching issue. I tried testing and switching from https://solana-api.projectserum.com which is the RPC endpoint I have been using, to a free node from QuickNode. This fixed the issue.

The problem is that this bot creates a lot of calls and is not able to be used with most free RPC's at all or for more than a day due to rates or amount of calls per month being limited.

To my knowledge, the only free public endpoint that isn't rate-limited is https://solana-api.projectserum.com. Is there a way to fix the RPC caching issue directly for the serum endpoint?

Thank you!

from liquidator-v3.

If you simply remove the ENDPOINT_URL environment variable it will default to using the Mango node, though this can get overloaded at times too. Shouldn't get caching problems there though.

Many liquidator operators are paying for private RPC for greater reliability.

from liquidator-v3.