theldus / 1brc Goto Github PK

Hi @Theldus!

First of all let me thank you for the very much enjoyable read that you gave me with this repo.

You did an excellent write-up and it was very nice to see all the changes discussed and their improvements!

Speaking of that, I have an idea on what is causing the speed up on v10, with the removal of the MAP_POPULATE flag.

Taken from man page of mmap:

MAP_POPULATE (since Linux 2.5.46)
Populate (prefault) page tables for a mapping. For a file
mapping, this causes read-ahead on the file. This will
help to reduce blocking on page faults later.

Since you are mapping a file, the MAP_POPULATE flag caused a prefault for the entire memory block. It is true that the memory will eventually be completely faulted (since the file will be read wholly), however the actual read is performed in threads, while the MAP_POPULATE flag happens in the main thread only.

So apparently, the concurrency of the page fault happening in the threads is causing the speedup.

What do you think about that?
Thanks again, wish you a wonderful day!