beling / bsuccinct-rs Goto Github PK

hi! nice work on these crates -- I'd like to use csf in my (commercial) project for building perfect hash tables in index files.

would you be open to licensing this repo with an MIT license?

Hi,

I'm trying to build a DynamicKeySet (actually, a CachedKeySet) from a rayon::iter::ParallelIterator instead of an Iterator. I feel this makes sense given I have a bunch of iterators as my own input, and fmph internally uses them as well.

I attempted to do this by copying the DynamicKeySet structure, and crudely replacing every : Iterator bound with a : ParallelIterator. This works for the most part; however I am stuck implementing for_each_key and into_vec as their predicate lack the Send and Sync bounds necessary to call rayon's .map().

And adding these bounds to the KeySet definition seems excessive, given that not all users need them.

Any tips?

Thanks!

We're trying to build an MPH for a dataset of 25B strings in the format swh:1:cnt:8cff7880b408c38231b723b4c568367912a33cb2 (about 1TB of data) from the Software Heritage graph. Previously we were using GOVMinimalPerfectHashFunction from Sux4J, which would construct the map in about 5h. However, after 15h we had to kill the ph process as it was still unfinished. Can you guide us into the best parameters/way of calling the lib to build such an MHP? Or is it such a long construction time typical?