Thank you for providing the informative article, DataFusion courtesy of Rust, vs Spark. Performance and other thoughts and code.

The article does not mention the source of the data, but I assume it's divvy-tripdata. Could you please elaborate on what data were used in your tests?

I've run the same Rust example, using the current DataFusion version, 32.0.0. The data files I used span from 202004-divvy-tripdata.csv through 202309-divvy-tripdata.csv, i.e 42 files.

I get a total run time of just over half a second (on a Macbook Prod M2):

...
...
| 57   | 2022.0 | 9.0   | 9.0  | Honore St & Division St                             |
+------+--------+-------+------+-----------------------------------------------------+
Elapsed: 670.41ms

For comparison, on the same laptop, I get a time of around 3 seconds from Spark.

Hence, I'm very surprised by the 180 second runtime that you got. Perhaps it is related to the earlier version of DataFusion that was used in your example. Would you mind running with the latest version of DataFusion?