blockchain storage & memory proposals about aeon-legacy HOT 11 CLOSED

Switching to Monero's LMDB is already in progress.

from aeon-legacy.

@iamsmooth is there a repo somewhere with the work in progress? I'm happy to help.

from aeon-legacy.

The impact is also that during saving to disk aeond doesn't respond to stratum-protol software, so miners get "pool is not responding", and that could be for up to several minutes. With big hashrate that is loss of income. Then, with eventual increase of chain length, it would be more and more demanding and devastating.

This 'feature' would impose a requirement for pools to run at least two nodes independently and keep care to run saving to disk at non-intersecting time. That's excessive burdensome.

Is it possible to make periodic "chekpoints" so that everything below that block is pushed out of memory? Thus keeping only last thousand or two blocks?
When comes a request to use all the chain, fork a separate process with minimal priority (and possible own cpu- and io- limits imposed by operating system) and do read-calculate-store in it?

from aeon-legacy.

The problem with current saving is that the whole chain is in-memory. Then when it's saved, the whole object is serialized and saved to a temp file which is then moved to the final destination. This whole process is slow and resource intensive.

Your periodic 'checkpoints' idea I think would require a fair amount of work but certainly another option for debate. I do like the idea of not having to hold the whole thing in memory.

Using a memory mapped file for the blockchain could solve both issues as it does address the loading and saving of very large files more efficiently. Boost has portable versions too which would work with the current serialization.

from aeon-legacy.

Why not to leave memory-specific issues to Operating System? It has really very good optimizations and could benefit from IO-operations with big cache, asynchronous reads/writes and so on and so on.
From sysadmin perspective, if aeond needs 'a quick file', then to speedup one could move it to SSD- or NVMe- storage, or to memory-mapped filesystem. That really works very fast.
Then aeond should just effectively read-write to a large file, and the rest takes care the OS.

from aeon-legacy.

Also, with this approach a sysadmin could get more metrics from filesystem to do bottleneck analysis, and measurably throw resources at it, comparing to a process which reads/writes to just vram.

BTW, if an OS decides that some pages in memory are not often necessary, it pushes them to swap.
Then all the program-side optimizations get ruined.
Instead, if the process just reads and writes to a file, the OS could speed up alot with its IO-cache, and not swapping the memory pages of a process.

from aeon-legacy.

Why not to leave memory-specific issues to Operating System?

Well that's the current implementation and that's the problem.

if an OS decides that some pages in memory are not often necessary, it pushes them to swap.

Again. Same issue. It's all slow.

Instead, if the process just reads and writes to a file, the OS could speed up alot with its IO-cache, and not swapping the memory pages of a process.

And that's where a memory mapped file could help, or refactoring to use something like LMDB or your 'checkpoints' suggestion and only loading what's needed in memory at any given operation.

Requirements will always require whole blockchain on disk and at least some, if not all, in memory at any given time. The question is what's the most efficient way to meet these requirements without a huge overhead on the system and causing other issues.

from aeon-legacy.

I do see that RDBMS like PostgreSQL can handle very sophisticated operations on large amount of data, obviously not placing it in memory as a whole thing. And relatively quick, and without blocking everything else from normal work.

I believe that these io-speed and data size bottleneck could be solved by invoking corresponding algorithms. To seek those, we need clarity in bottleneck sources, that is why @jtgrassie, would you please describe in more detail the algorithm of current saving the blockchain implementation?

from aeon-legacy.

Hi. I just installed AEON on an old but well working machine: I really can't use it as downloading the blockchain to memory makes my computer useless.

from aeon-legacy.

@iamsmooth me too

from aeon-legacy.

Bounty open and discussed on reddit and bitcointalk thread

from aeon-legacy.