One Billion Row Challenge
Attempts at the 1 Billion Row Challenge.
To get started pixi run create_measurements to generate 1 billion city temperature pairs (approximately 13 GB).
Environments and tasks are managed with Pixi for sanity.
| time (seconds) | pixi run <task> |
description |
|---|---|---|
| 776.58 s | py_1_std_lib |
Just using the standard library (plus TQDM to keep track) and iterating over a file by lines |
| 129.07 s | py_2_pandas |
Pandas groupby and aggreggate |
| 56.22 s | py_3_dask |
Dask groupby and aggregate |
| error (locked up machine) | py_4_polars_eager |
Using Polar's eager or immediate Pandas-like API caused my computer to lock up |
| 130.02 s | py_5_polars_lazy |
Polars has a lazy API, though it wasn't much faster (though lower memeory) than Pandas as it couldn't skip reading data. |
| 9.27 s | py_6_1_ibis_duckdb |
Ibis is a library that wrapps multiple ways of interacting with dataframes. The default way is to use DuckDB, which managed to read the file at around 1.5 GB/s! |
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent