oasysai / oasysdb Goto Github PK
View Code? Open in Web Editor NEWHybrid vector database with flexible SQL storage engine & multi-index support.
Home Page: https://docs.oasysdb.com/
License: Apache License 2.0
Hybrid vector database with flexible SQL storage engine & multi-index support.
Home Page: https://docs.oasysdb.com/
License: Apache License 2.0
We should add an API for the Collection
called maybe filter
that takes in the condition and return a hash map of VectorID to Record. This filter API is useful for users who need to retrieve some of records which data match the provided condition.
Currently, there is no way to reset all of the value stored in the database. This feature is a good to have feature for users to explore inserting values to OasysDB. This feature will allow a quick reset via API without replacing Docker container or removing the data storage manually.
Users should be able to call an endpoint: DELETE /values
and DELETE /graphs
. Each endpoint will be responsible to reset the data stored in each database. I think this is better than having one endpoint to reset both database as this approach allows a more fine-grained control over the database.
An alternative solution that I thought about is to have one utility endpoint like /reset
to remove all of the database storage from disk. One of the issue with this approach is that it's less RESTy which what we are trying to fully implement with OasysDB.
This bug happens when saving a collection in a periodic manner like the example below.
# Create the DB and collection...
for _ in range(300):
records = Record.many_random(dimension=512, len=100)
collection.insert_many(records)
db.save_collection("storage", collection)
This caused unflushed dirty IO buffers to bloats the database folder to an amount unimaginable ๐คฏ
From 70MB database size to 10GB.
Read the discussion #77 for more context about what is happening.
This new API should allow users to access all of the records inside of the collection to perform operation on the record beyond the capability of the database such as filtering or iterating.
This issue is correlated with PR #40.
I'm trying to integrate OasysDB as part of a rust staticlib, which gets linked to a C++ based project. I'm not using Python integration at all, and yet all of pyo3
dependencies are compiled and the final binary needs to be linked with python library.
Please make pyo3 an optional feature.
In OasysDB, currently, there is no dimension checks when inserting new vectors.
This could lead to unchecked mistake by users.
In the IndexParams trait, we should add a method:
fn dimension(&self) -> usize;
This method will return integer of the configured dimension of the index. When building and inserting vectors in each index implementation, we need to add a check to ensure the vector dimension is valid.
This chore purpose is to refactor the database (src/db/database.rs
) public methods to include a more robust error handling functionality. This includes changing the return of the functions to add Result
.
// Current return type on create_collection method.
Collection<D, N, M>
// Expected return type.
Result<Collection<D, N, M>, Box<dyn Error>>
This chore will help users use OasysDB as a crate as it will help with error handling and debugging. This also creates a base of a more robust error messages for OasysDB in general along with issue #24 .
OasysDB only has Rust generated documentation that is hosted in docs.rs/oasysdb. Not everyone who uses OasysDB uses Rust and thus would be unfamiliar with the documentation.
With dedicated documentation website for OasysDB, users should be able to understand and use OasysDB better.
This chore purpose is to improve error handling of the collection public functionality to include error as value and use Result
as returns. The scope of this chore is limited to the src/func/collection.rs
file to avoid huge PR.
The result of this chore should look like this code below:
// Old returns
Vec<SearchResult<D>>
// New returns
Result<Vec<SearchResult<D>>, Box<dyn Error>>
This chore/improvement will help greatly when users use OasysDB as vector database. This chore helps make error handling with OasysDB much more intuitive.
When calculating Cosine similarity on normalized vectors there is no need to calculate the magnitudes of each vector as they are already of unit length.
OpenAI embeddings are normalized, and presumably many others are as well. From the OpenAI embeddings FAQs:
OpenAI embeddings are normalized to length 1, which means that:
- Cosine similarity can be computed slightly faster using just a dot product
The Cosine distance calculation from distance.rs#L46 could benefit from this optimization and avoid the last 3 calculations:
fn cosine(a: &Vector, b: &Vector) -> f32 {
let dot = Self::dot(a, b);
let ma = a.0.iter().map(|x| x.powi(2)).sum::<f32>().sqrt();
let mb = b.0.iter().map(|y| y.powi(2)).sum::<f32>().sqrt();
dot / (ma * mb)
}
I suggest the Distance
enum be expanded to include a CosineOptimizedUnitLength
variant. Doing so would fit in nicely with the current config:
let mut config = Config::default();
// Using optimized calculation as our embeddings are normalized
config.distance = Distance::CosineOptimizedUnitLength;
The Distance::calculate
function could then match on this and call an optimized method:
fn cosine_normalized(a: &Vector, b: &Vector) -> f32 {
Self::dot(a, b)
}
Not applicable / did so already inline where appropriate.
When inserting a vector record to a collection, users need a way to get the newly inserted record vector ID. When users need to update or delete the vector records, this returned vector ID can be used as the input parameters to the methods.
Modify the return type and value of the Collection.insert
method.
// Before
-> Result<(), Error>
// After
-> Result<VectorID, Error>
OasysDB uses Euclidean distance formula to measure the distance between vectors by default. Some use cases of the users might require them to use other distance metrics like Dot Product or Cosine Similariy.
I think the best solution would be to allow users to configure the distance formula choice in the Collection configuration level. Something like:
pub struct Config {
...
distance: String
}
Then after that, in the Vector
implementation:
pub fn distance(&self, other: &Self, formula: &str) -> f32 {...}
This feature should create a script that allows us to measure the recall rate of OasysDB. This is very useful to make sure that when developing new features and refactoring, we don't decrease the recall performance of the database as this is a very important metric.
Got interested in that project. I am a rust beginner and know a little about vector database and how hnsw is implemented. :)
Is it free to take part in and Any good first issue for beginner? Hope to write some code for this project.
This feature should add a new index implementation for HNSW indexing algorithm.
This allows users to configure the index to use this algorithm.
idx_hnsw.rs
file.This is a request to add several utility methods to help users use the VectorID
struct better. These are some ideas for improvement:
VectorID
to usize
and u32
.to_usize
and to_u32
.Allows users to store boolean as metadata or most commonly object containing boolean values like:
{
"text": "...",
"is_private": true,
}
Just add boolean to metadata enum and adds type conversion to/from JSON, PyO3, and Rust native types.
Hi, I'm using Oasysdb 0.5.1
I got a panic error when I called insert
method of collection:
thread 'dialog-flow-system' panicked at D:\programData\rust\cargo\registry\src\index.crates.io-6f17d22bba15001f\oasysdb-0.5.1\src\func\utils.rs:413:14:
called `Option::unwrap()` on a `None` value
stack backtrace:
0: std::panicking::begin_panic_handler
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\panicking.rs:645
1: core::panicking::panic_fmt
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\core\src\panicking.rs:72
2: core::panicking::panic
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\core\src\panicking.rs:145
3: core::option::unwrap_failed
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\core\src\option.rs:1985
4: enum2$<core::option::Option<usize> >::unwrap
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\option.rs:933
5: oasysdb::func::utils::IndexConstruction::insert
at D:\programData\rust\cargo\registry\src\index.crates.io-6f17d22bba15001f\oasysdb-0.5.1\src\func\utils.rs:410
6: oasysdb::func::collection::Collection::insert_to_layers
at D:\programData\rust\cargo\registry\src\index.crates.io-6f17d22bba15001f\oasysdb-0.5.1\src\func\collection.rs:748
7: oasysdb::func::collection::Collection::insert
at D:\programData\rust\cargo\registry\src\index.crates.io-6f17d22bba15001f\oasysdb-0.5.1\src\func\collection.rs:240
I created a simple repository and tried reproducing this panic, but I can't get the error again
repo: https://github.com/dialogflowchatbot/oasysdb
After I deleted the data directory (The one maybe cause the panic) , the error was gone.
Do you have any thoughts about the reason cause this panic?
Add an example to set up and use OasysDB with PostgreSQL.
This allows PostgreSQL users to get started with OasysDB much easier.
Currently, the database data, key-value pairs and index, is stored only in memory. This means that every time the database is restarted, we need to re-add the key-value and rebuild the index.
This feature should allow the database data to be persisted to disk so that when there is crash or shutdown, the data and index of previous session can still be used.
The database should store a snapshot of the key-value pairs periodically. Every time, the database is started, the database will lookup the most recent snapshot and prepopulate the key-value store and rebuild the index based on the existing key-value pairs.
This feature will add a utility connector that streamline insert, update, and search operations of OasysDB allowing users to directly perform these operations without converting the content to vector embeddings beforehand utilizing the EmbeddingModel
trait.
Example:
// Before
...
let vector: Vector = generate_vector(content);
let record = Record::new(&vector, &data.into);
collection.insert(&record);
// After
collection.insert_content(content, &data.into);
insert_content
, search_content
, etc.gen
.pub fn insert_content(content: &str, data: &Metadata) ...
// All vector methods with suffix _content or for batch operations _contents.
// The output of the method should be the same with their default couterparts.
The scope of this feature doesn't include accomodating third-party embedding model. We will do it in the future though depending on the demand.
This change should be backward compatible if possible.
A lot of applications created in Rust uses justerror
or anyhow
crates to handle errors. Currently, it's not intuitive to use OasysDB built-in Error
struct with other commonly used error handling crates. We should add Display
implementation to Error
so that it's easier to wrap the built-in error with these crates.
Users need a way to insert a lot of vector records at once outside of the Collection.build
method.
Technically, users can use Collection.insert
method combined with a loop statement. But, this is highly inefficient because the private collection method, insert to graph layers, is designed to insert one vector record at a time.
We might need a new private method dedicated to inserts multiple vector records to the index layers at once. We don't want to use for loop combined with insert_to_layers
method as it is highly unoptimized for batch inserts.
Most of the use cases of vector databases seems to be implemented in Python as it has a more mature AI ecosystem than Rust. For example, Chroma allows their users to use its vector database simply by installing its Python libraries.
Some LLM frameworks like LangChain is also widely used when implementing LLM-related functionality which typically includes a form of vector stores. By having a Python binding to OasysDB, we open the door for this integration.
The users should be able to install OasysDB using pip install oasysdb
command. Also, all of the functionality required to run the vector database should be accessible using Python. For this, I found a library that can help with creating Rust to Python binding: https://github.com/PyO3/pyo3.
Ran Database::open
failed
Version of Oasysdb
: 0.7.1
use std::path::Path;
let p = Path::new(".").join("data").join("intentev"); // relative path to D:\work\dialogflow-backend\data
Database::open(p, Some("sqlite://./data/intentev/e.dat"))?;
open
function throws
code: FileError, message: "The system can't move files to a different disk drive. (os error 17)"
I suspected that this error was thrown by utils/file.rs
at this line: fs::rename(&tmp_file, &path)?;
I printed temporary directory on Windows was: C:\Users\dialogf\AppData\Local\Temp\
and Datebase
path I specified was: D:\work\dialogflow-backend\data\intentev\
Database open successfully.
Add any other context about the problem here like screenshots or logs.
Check out PR #63.
Updated from oasysdb 0.4.4 to 0.5.0. Building my program now gives this error:
--> ~/.cargo/registry/src/index.crates.io-6f17d22bba15001f/oasysdb-0.5.0/src/db/mod.rs:9:17
|
9 | use std::hash::{DefaultHasher, Hash, Hasher};
| ^^^^^^^^^^^^^ no `DefaultHasher` in `hash`
|
= help: consider importing this struct instead:
std::collections::hash_map::DefaultHasher
For more information about this error, try `rustc --explain E0432`.
error: could not compile `oasysdb` (lib) due to previous error
Compiling on a M1 MacBookPro.
I wanted to do sentence similarity
git clone https://github.com/dialogflowchatbot/oasysdb-test
cd oasysdb-test
git lfs install
git -C resources clone https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2
cargo r
I'm using rust-bert to generate sentence embedding, so the modal files need to be downloaded first.
The 1st searching, won't match anything
How severe is the bug?
Normal
Are you using OasysDB in production?
Not yet, but planing.
Add any other context about the problem here. For example, screenshots, screen recordings, or logs.
Currently, the relevancy score is applied after the search result has been retrieved. This could cause some other nodes that are more relevant to be left out due to the nature of the index.
We want to apply the relevancy score during the creation of the candidate pool.
This feature should add a flat IVF index implementation that excludes IVFPQ. IVF can offer a way higher recall rate than IVFPQ due to the loss that happens during the encoding and decoding process of PQ vector.
idx_ivf.rs
.Adding filtering based on metadata when performing a search. This gives a better search result due to the nature of predeterministic of the filter that ANN doesn't provide.
This implementation must be pre-filtering. This means the filtering happens not after the result of ANN but during the creation of nearest neighbors candidate pool.
When users are dealing with complex data structure such as that of serde_json::Value
, it requires a really complex workaround to make it work well with Metadata
even though they share the same model.
OasysDB's Metadata
and serde_json::Value
should be able to convert into one another by using the From and Into implementations.
The bug happens when running cargo test
in the terminal. Reason: no LC_RPATH's found.
The bug happens after following this steps below from nothing:
(Running on VS Code, MacOS 14.2 M2 Chips, Python Version installed 3.11.8)
cargo test
from the terminal.Expected: The compilation works successfully and giving no errors
Error encountered: Library not loaded: @rpath/Python3.framework/Versions/3.9/Python3, error: test failed, to rerun pass --lib
.
This might blocks the new users or contributors to use or contribute.
Need target support for wasm32-unknown-unknown for developing on ICP ecosystem. Getting this error currently when building, something to do with the simsimd crate -
Caused by: Failed while trying to install all canisters.
Caused by: Failed to install wasm module to canister 'elna_db_backend'.
Caused by: Failed during wasm installation call
Caused by: The replica returned a rejection error: reject code CanisterError, reject message Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai: Canister's Wasm module is not valid: Wasm module has an invalid import section. Module imports function 'simsimd_cos_f32' from 'env' that is not exported by the runtime..
This is likely an error with the compiler/CDK toolchain being used to build the canister. Please report the error to IC devs on the forum: https://forum.dfinity.org and include which language/CDK was used to create the canister., error code None
Here's the relevant error from above - Module imports function 'simsimd_cos_f32' from 'env' that is not exported by the runtime..
Please support wasm32-unknown-unknown as a target.
When users are using anyhow
crate to handle error on their Rust project, there is no straight-forward solution to cast OasysDB native error type to anyhow error beside using .map_err()
method. Although it works, when handling multiple errors from OasysDB, users have to write more boilerplate codes to handle errors.
Currently, when performing Database::save_collection
method, OasysDB serialize the collection data into binary and store it directly into a file. When dealing with a large file that requires multiple second to write, the risk of a file being corrupted if the write process is interrupted is high.
We want to make save_collection
operation atomic by writing the serialized collection into a temporary file before replacing the actual collection file. This way, if both write and replace operation fails, the actual collection file won't be corrupted.
After upgraded from 0.3.0 to 0.4.1, codes break.
ImportError
Traceback (most recent call last) Cell In[1], line 1
----> 1 from oasysdb.prelude import Vector, Config, Database, Collection, Record
File ~/.pyenv/versions/3.11.4/envs/env/lib/python3.11/site-packages/oasysdb/__init__.py:2
1 # flake8: noqa F401
----> 2 from .oasysdb import *
ImportError: dynamic module does not define module export function (PyInit_oasysdb)
Just import from oasysdb.preludes
Works like a charm
BTW, great work.
Add a script to benchmark the database performance, the result of the benchmark, and the documentation of the benchmarking process. The primary operation that we want to have the benchmark for is the graph querying operation.
This is good to help users compare OasysDB to other vector database and make decision. This also helps us improve OasysDB by comparing the performance benchmark to benchmark making sure we do something that improve the database.
Currently, the Collection.search
method will return the approximate nearest neighbors regardless of the distance. There is a possibility that the result is totally irrelevant to the query but is still included in the search result regardless. Refer to issue #46.
This feature should allow users to configure the collection so that if the result candidate distance doesn't pass the configured threshold, the candidate won't be included in the search result.
This makes sure that only relevant result candidates is included in the search result.
This value should be included in the Config
struct or directly in Collection
struct.
One of the potential roadblock is how to make this configuration more intuitive. Distance value will differs depending on the distance formula. We could use arbitrary float value but that might requires the user to readjust the value.
Hi,
Just visited https://www.oasysai.com/ and browser said the certificate was expired.
Websites prove their identity via certificates, which are valid for a set time period.
The certificate for www.oasysai.com expired on 4/23/2024.
I found certificate was from Let's Encrypt
, maybe can use renew helper to obtain certificate automatically.
BTW, if here's not a proper place to report this, please delete this thread as you wish.
This feature should allow users to configure Manhattan distance metric for their index:
Manhattan Distance
Improve the performance when reading and writing collection from/to the disk.
The scope of this optimization is backward compatibility.
Ultimately, to improve the speed, we need to reduce the overall collection size by off loading the storage of the vectors and metadata to disk permanently.
Use bincode::serialize_into()
and bincode::deserialize_from()
.
We need a functionality to measure the memory usage of the collection and a documentation about the memory usage when storing and searching the collection. For example, how much memory is required to store and query SIFT datasets.
This functionality and documentation of memory usage measurement will help users evaluate using OasysDB and understand the requirements for their use case better.
Relevant resources:
Love the project! Would love an integration with Indexify - https://github.com/tensorlakeai/indexify
We are building an online extraction and retrieval engine for any type of unstructured data. It has a pluggable interface for vector stores - https://github.com/tensorlakeai/indexify/blob/main/src/vectordbs/mod.rs
Content uploaded into Indexify can then be queried using SQL and Vector Search!
Any thoughts?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.