Make transaction processing more scalable about graham_discord_bot HOT 3 CLOSED

Ok, here's my proposed solution. Would love to hear your thoughts on this. We can introduce a separate process that handles sending and receiving, and is backed by Redis/Sidekiq.

1. TipBot transaction is created. We store the transaction in SQLite like we already do.
2. After writing the transaction to the db, we build a Sidekiq job and push it into the send queue. Doing so is trivial from Python.
3. Over in the new worker process, the job is picked up and the transaction is sent to the node RPC.
4. Once the transaction is successfully sent, the job schedules a corresponding receive job for the destination address with the address and block in the payload.
5. The receive job is picked up by the worker, and it tells the node RPC to receive the block for the account. This is idempotent, so if the RPC fails for whatever reason, it will be automatically retried until it succeeds.

This solution is highly scalable because you can increase the number of workers and threads as needed. As the workers scale, the bottleneck will become the node, so we'll have to handle that separately if needed (but probably not for awhile).

One missing piece is that we probably want a way to update the database and tell it the transaction has been processed. Might require moving to Postgres, or we can set up some kind of interprocess communication between the two.

from graham_discord_bot.

Sounds about perfect to me. Besides not really needing to worry about the receives being retried. We don't care about the result of that job as much as we do the result of the send job .

Sending could use the retry mechanism, and an easy solution to the sqlite multi processing problem is probably just switching to postgres so maybe issues #3 and #5 could be prerequisites to this one.

from graham_discord_bot.

@renesq I need some support (paid also fine) for an issue, can i have your email ? or can you drop me contact on [email protected]

from graham_discord_bot.