Lazy / Cached Spill Over to Disk about questdb HOT 7 CLOSED

NFSdb bottleneck is not with disk. Memory Mapped files are perfectly efficient to achieve more than NFSdb does. Speed is a traid off of functionality. Index, partitioning and object reflection take most of the time.

Also making journal not make mapped files and than spill over will be very inefficient.

from questdb.

May be you can option that avoids indexing and partitioning incase you just write and read from the journal as a queue. (With auto commit.) A client can just read it and pipe it into another journals each being a queue based on an allocation methodology and applying a filer. Similarly you can pipe into multiple journals which the clients can read independently.

I am not sure how IO scales with multiple journals but will this be faster? At Least if might be functionally superior by having messaging patterns (#14).

from questdb.

Partitiioning and indexing are fully optional and both are off by default. You can try to see if performance suits your use case. I'll cover auto-commit under issue #9.

BulkWriter and BulkReader work at about 95% of performance of non-bulk variants and provided they only do sequential writes/reads they are not affected by your RAM size even. They'll work at constant rate until you run out of disk space. I think they cover the spill over problem in most efficient way possible albeit at a slight overall performance loss.

from questdb.

there is a major new functionality coming in version 3.0, which may have an influence on subject of in-memory only store. Let keep this closed until than to keep backlog clear.

from questdb.

OK. The only part may be an issue.

from questdb.

Do tell more :)

from questdb.

What you run out of memory. Can you spill into disk?

from questdb.