Synthetic table data about locustdb HOT 6 CLOSED

I usually use mockeroo for generating data.
Sakila is also commonly used in opensource database apps as sample.
I've also gathered data from various sources and put them here.

I've also written some webapp that showcase displaying these database.

from locustdb.

Some properties we probably want from our data generator:

• able to run in-process and without reading from disk
• generate 100M values within seconds
• deterministic with fixed seed
• can be extended with new data distributions

from locustdb.

I think you guys are looking for meaningful data rather than just noise, but SeaHash might be of some use. You can define your own seed for it. The following code iterates 100 million times in just under a second on my machine.

extern crate seahash;

use seahash::SeaHasher;
use std::hash::Hasher;

fn main() {
    let mut hash = SeaHasher::new();
    let mut last = 0;

    for _ in 1..100_000_000 {
    	hash.write_u64(last);
    	last = hash.finish();	
    }

    println!("{:#X}", last);
}

from locustdb.

Actually, completely random and incompressible data is an important test case so SeaHash could be a very useful basis for one the generators.

from locustdb.

09467c6 implements basic framework for random column generation, as well as specific generators for int and string columns based on Markov chains which should capture most properties we care about with regards to performance (except correlations between columns).
Still worthwhile to add additional simpler distributions that are easier to configure and faster to generate.

from locustdb.

f8688fe adds composable generator that picks a portion of partitions and replaces them with all null values and #41 adds generators for random strings.

from locustdb.