0xdebabrata / citrus Goto Github PK

Integrate sqlite to store plain text and other supplied metadata.

• #16 adds a fallback response which returns all vectors for a particular index.
• Filter the response based on the user-provided filters.

By default, indices on citrusdb are of 1536 dimensions. Expose APIs to allow developers to set any vector dimension during index creation.

citrus only supports integer IDs at the moment because that's passed directly to HNSW. Add support for non-integer IDs by creating a map between each ID and an incrementing integer value which will serve as the HNSW label.

Currently citrus supports vector embeddings from OpenAI text-embedding-ada-002 model.

It'd be great to add support for other embedding models from Cohere, Huggingface, etc. Create an EmbeddingFunction abstract class or something similar and implement OpenAI, Cohere, etc. embedding functions.

The create_index function no longer takes persist_directory in the signature and now requires name.

See README:
https://github.com/0xDebabrata/citrus/tree/main#1-create-index

add now requires index which matches the value for name in create_index.
See README:
https://github.com/0xDebabrata/citrus/tree/main#2-insert-elements

Allow reloading HNSW indices from index using data stored in index_manager table.

This would allow citrus to automatically load existing indices from disk during server restarts and take us one step closer to offering a hosted cloud solution.

Currently, all indices need to be in memory to perform CRUD operations on a given index that has been saved to disk and currently not in memory.
This creates unnecessary latency and memory usage as all indices are queried from sqlite before being loaded into memory.
Moreover the get_all_indices method also returns a count of elements in all indices. This query runs very slowly for large indices.

Add a new method that reloads just the required index to memory.