Iterating through larger data use too much memory and performance is low about lisk-db HOT 1 CLOSED

After consideration of different options/optimizations regarding pub fn js_iterate(mut ctx: FunctionContext) -> JsResult<JsUndefined> function to make it less memory intensive, we decide to leave it as it is.

Current Neon implementation has some limitations on how you can pass borrowed objects back to JS and this was one of the main issues, why memory optimization was not done.

We also investigated how napi-rs library works with lisk-db and measurements showed that a peak of a memory consumption was decreased by about 50%.
We decided to not proceed and use this library instead of neon because an effort in comparison to what we will be gaining would be too big.
High memory consumption only concerns lisk-migrator because it iterates over a large number of database elements. Lisk nodes are not affected by this because iterations are much smaller in comparison with lisk-migrator.

from lisk-db.