azure_kusto_vector's Introduction

Kusto as a Vector database (Eventhouse in Microsoft Fabric/Azure Data Explorer)

The following section is also applicable to Eventhouse in Microsoft Fabric

Azure Data Explorer aka Kusto is a cloud-based data analytics service that enables users to perform advanced analytics on large datasets in real-time. It is particularly well-suited for handling large volumes of data, making it an excellent choice for storing and searching vectors. Kusto is also available in

Kusto supports a special data type called dynamic, which can store unstructured data such as arrays and properties bag. Dynamic data type is perfect for storing vector values. You can further augment the vector value by storing metadata related to the original object as separate columns in your table.
Kusto also supports in-built function series_cosine_similarity_fl to perform vector similarity searches.

Get started with Kusto for free.

Getting started with Kusto and Open AI embedding

Demo Scenario

If you’d like to try this demo, please follow the instructions in the Notebook.

It will allow you to -

Use precomputed embeddings created by OpenAI API.
Store the embeddings in Kusto.
Convert raw text query to an embedding with OpenAI API.
Use Kusto to perform cosine similarity search in the stored embeddings.

You can check the solution with AAD authentication and Kusto queries here.

azure_kusto_vector's People

Contributors

Stargazers

Watchers

azure_kusto_vector's Issues

Error: KustoApiError: Request is invalid and cannot be processed: Semantic error: SEM0260: Unknown function: 'series_cosine_similarity_fl'

Cosine function syntax error

About this function in the notebook:

.create-or-alter
series_cosine_similarity_fl(vec1:dynamic, vec2:dynamic, vec1_size:real=double(null), vec2_size:real=double(null))
{
    let dp = series_dot_product(vec1, vec2);
    let v1l = iff(isnull(vec1_size), sqrt(series_dot_product(vec1, vec1)), vec1_size);
    let v2l = iff(isnull(vec2_size), sqrt(series_dot_product(vec2, vec2)), vec2_size);
    dp/(v1l*v2l)
}```

This statement causes a syntax error. I believe the correct statement is this:

.create-or-alter function series_cosine_similarity_fl(vec1:dynamic, vec2:dynamic, vec1_size:real=double(null), vec2_size:real=double(null))
{
    let dp = series_dot_product(vec1, vec2);
    let v1l = iff(isnull(vec1_size), sqrt(series_dot_product(vec1, vec1)), vec1_size);
    let v2l = iff(isnull(vec2_size), sqrt(series_dot_product(vec2, vec2)), vec2_size);
    dp/(v1l*v2l)
}

Can you please verify?

Recommend Projects

anshulsharmas / azure_kusto_vector Goto Github PK

azure_kusto_vector's Introduction

Kusto as a Vector database (Eventhouse in Microsoft Fabric/Azure Data Explorer)

The following section is also applicable to Eventhouse in Microsoft Fabric

Getting started with Kusto and Open AI embedding

Demo Scenario

You can check the solution with AAD authentication and Kusto queries here.

azure_kusto_vector's People

Contributors

Stargazers

Watchers

Forkers

azure_kusto_vector's Issues

Error: KustoApiError: Request is invalid and cannot be processed: Semantic error: SEM0260: Unknown function: 'series_cosine_similarity_fl'

Cosine function syntax error

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent