sailfish009 / granne Goto Github PK

granne (graph-based retrieval of approximate nearest neighbors) is a Rust library for approximate nearest neighbor search based on Hierarchical Navigable Small World (HNSW) graphs and is used in Cliqz Search. It focuses on reducing memory usage in order to allow indexing billions of vectors.

• Memory-mapped
• Multithreaded index creation
• Extensible indexes (add elements to an already built index)
• Python bindings
• Dense float or int8 elements (cosine distance)

You will need to have Rust installed. This can be done by calling:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Or by visiting https://rustup.rs/ and following the instructions there.

# build
cargo build --release

# test
cargo test

# bench
cargo +nightly bench

See Python Bindings.

granne can use BLAS (https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) to improve speed of some computations. On Debian/Ubuntu both libblas-dev and libopenblas-dev should work, with the latter being significantly faster.

BLAS can be enabled by passing the blas feature during compilation, e.g.

cargo build --release --features "blas"

On Mac OS there seems to be some issue (maybe this one) with the default BLAS library. A workaround is to install e.g. openblas and link to that instead.

*granne is Swedish and means neighbor