xycloo / rs-ingest Goto Github PK

Currently, the runner creates unbounded channels with std::sync::mpsc::channel(). This means every send op goes through in the bufreader since the channel has an infinite buffer and might cause overloads if the streamed data from code overloads the decoder. While this might be what most users look for, adding the option for bounded channels might help in some situations to prevent excessive memory usage.

When running single-threaded the closing mechanism (which is not yet implemented) could be within the runner core module. However, when running multi-threaded the closing mechanism can't be triggered by rs-ingest modules since it would halt execution.

The best way forward seems to be to provide a handle users should call to initiate closing.

When running catchup in multi-thread mode users might incur an error within the MetaResult wrapper since after catching up it will try to decode the catchup result. The most straightforward way to approach this is to document this and notify users not to do unsafe access to options of MetaResult since it will always result in a panic when running catchup in multi-thread mode.

rs-ingest currently seems to work well also on pubnet and a new flag in IngestionConfig has been added to specify wether to run on mainnet or futurenet.

That said, I haven't had the chance yet to see how it goes when running online or going through big chunks of history. We should also document pubnet compatibility and add examples, possibly similar to the already-existing ones on stellar/go/ingest.