asking for documentation of usage and more detailed instructions about locustdb HOT 3 CLOSED

I think after reading comments you have made under posts, I can conclude that the size of RAM depends on compression ratio and data size.

Let's say I have TBs of data and I only have machines(several) with 32GB RAM, is there any way I can speed up the exploration of data? LocustDB can fit in here? Or some other alternatives could be considered?

from locustdb.

Ah, you beat me to it :). Right now LocustDB does not support querying data across multiple nodes. In principle it is architected to make this possible, but it's still a lot of work to actually implement. If you are looking for a system you can use in production right now, you may want to look at e.g. ClickHouse.

Answer to original question:

Depends on compression. LocustDB actually doesn't really apply much compression yet beyond dictionary encoding strings and choosing the appropriate byte-width for integer columns.

For the taxi dataset, the uncompressed csv takes up about 600GB, the 120GB number is after applying gzip. I also dropped about 70% of columns when loading the dataset into LocustDB, without that you would need maybe 150GB. So in this case, compression is similar to gzip and around ~4x over uncompressed csv. I think those numbers are somewhat typical, but will depend on the actual dataset.

If we implement additional compression passes, it should be possible to get compression ratios in excess of 10x (but sacrificing some query speed). Adding support for storing data on disk would also help reduce memory requirements, but performance will suffer quite significantly if the part of the dataset that you query doesn't fit completely into memory.

from locustdb.

Thanks a lot for the detailed answer. I guess I can just close this issue. Will do some other research.

from locustdb.