Suggestion: Use the Rust implementation of xxHash about huniq HOT 6 CLOSED

XXHash has been implemented in rust; XXH3 (which is a lot faster) has not… Can you point me at a XXH3 implementation in rust?

from huniq.

There is a PR for the crate mentioned above implementing XXH3 in Rust.

I would like to use huniq as a cross-platform uniq, that I can easily install with cargo. Maybe I will just have to write my own inspired by your post in Medium.

from huniq.

I just had a look at the code of huniq. It seems to me that it would not require much effort to offer the possibility to choose between the Rust implementation of XXHash and the bindings to the C implementation of XXH3. That way people who are willing to sacrifice performance for ease of installation can benefit from your effort too :)

from huniq.

That is great! We can try to move to that once the PR is merged. I would be happy to accept a PR that adds a feature flag!

from huniq.

Imo, a feature flag can make the rust implementation much less useful, since most users will do cargo install huniq, and still use xxhash. Assuming that benchmarking (on diverse data, with various amounts of lines, length distribution and ordering) shows that using twox-hash is at least as fast as xxhash, would you consider accepting a PR that completely replaces xxhash ?

from huniq.

twox implements xxhash, this uses xxh3 which are separate algorithims…
Benchmarks have shown that xxh3 is quicker in one shot mode, but at this point I am ready to question the validity of those benchmarks…

The benchmarks need to be redone, this time in proper rust…

I'd say if xxh3 is to be replaced, it should be replaced by ahash not twox…

from huniq.