Giter Club home page Giter Club logo

vector-js's Introduction

Upstash Vector Node.js Client · license Tests npm (scoped) npm bundle size npm weekly download

Note

This project is in GA Stage.

The Upstash Professional Support fully covers this project. It receives regular updates, and bug fixes. The Upstash team is committed to maintaining and improving its functionality.

@upstash/vector is an HTTP/REST based client for Typescript, built on top of Upstash REST API.

It is the only connectionless (HTTP based) Vector client and designed for:

  • Serverless functions (AWS Lambda ...)
  • Cloudflare Workers
  • Next.js, Jamstack ...
  • Client side web/mobile applications
  • WebAssembly
  • and other environments where HTTP is preferred over TCP.

See the list of APIs supported.

Quick Start

Install

Node.js

npm install @upstash/vector

Create Index

Create a new index on Upstash

Basic Usage:

import { Index } from "@upstash/vector";

type Metadata = {
  title: string,
  genre: 'sci-fi' | 'fantasy' | 'horror' | 'action'
  category: "classic" | "modern"
}

const index = new Index<Metadata>({
  url: "<UPSTASH_VECTOR_REST_URL>",
  token: "<UPSTASH_VECTOR_REST_TOKEN>",
});

//Upsert data
await index.upsert([{
  id: 'upstash-rocks',
  vector: [
    .... // embedding values
  ],
  metadata: {
    title: 'Lord of The Rings',
    genre: 'fantasy',
    category: 'classic'
  }
}])

// Upsert data as plain text.
await index.upsert([{
  id: 'tokyo',
  data: "Tokyo is the capital of Japan.",
}])

//Upsert data alongside with your embedding
await index.upsert([{
  id: 'tokyo',
  data: "Tokyo is the capital of Japan.",
  vector: [......]
}])


//Query data
const results = await index.query<Metadata>(
  {
    vector: [
      ... // query embedding
    ],
    includeVectors: true,
    includeMetadata: true
    topK: 1,
    filter: "genre = 'fantasy' and title = 'Lord of the Rings'"
  },
  {
    namespace: "example-namespace"
  }
)

//Query with your data
const results = await index.query(
  {
    data: "Where is the capital of Japan",
    topK: 1,
  },
)

//Query with your data
const results = await index.query(
  {
    vector: [
      ... // query embedding
    ],
    includeData: true, // Returns data associated with your vector.
    topK: 1,
  },
)

// If you wanna learn more about filtering check: [Metadata Filtering](https://upstash.com/docs/vector/features/filtering)

//Update data
await index.upsert(
  {
    id: "upstash-rocks",
    metadata: {
      title: 'Star Wars',
      genre: 'sci-fi',
      category: 'classic'
    }
  },
  {
    namespace: "namespace"
  }
);

//Delete record
await index.delete("upstash-rocks", {namespace: "example-namespace"});

//Delete many by id
await index.delete(["id-1", "id-2", "id-3"]);

//Fetch records by their IDs
await index.fetch(["id-1", "id-2"], {namespace: "example-namespace"});

//Fetch records by their IDs
await index.fetch(["id-1", "id-2"], {namespace: "example-namespace", includeData:true});

//Fetch records with range
await index.range(
  {
    cursor: 0,
    limit: 5,
    includeVectors: true,
  },
  {
    namespace: "example-namespace"
  }
);

//Reset index
await index.reset();

//Info about index
await index.info();

//Random vector based on stored vectors
await index.random({namespace: "example-namespace"});

//List existing namesapces
await index.listNamespaces();

//Delete a namespace
await index.deleteNamespace("namespace-to-be-deleted");

Namespaces

Upstash Vector allows you to partition a single index into multiple isolated namespaces. Each namespace functions as a self-contained subset of the index, in which read and write requests are only limited to one namespace. To learn more about it, see Namespaces

Example

import { Index } from "@upstash/vector";

type Metadata = {
  title: string;
  genre: "sci-fi" | "fantasy" | "horror" | "action";
  category: "classic" | "modern";
};

const index = new Index<Metadata>({
  url: "<UPSTASH_VECTOR_REST_URL>",
  token: "<UPSTASH_VECTOR_REST_TOKEN>",
});

const namespace = index.namespace("example-namespace");

//Upsert Data
await namespace.upsert([{
  id: 'upstash-rocks',
  vector: [
    .... // embedding values
  ],
  metadata: {
    title: 'Lord of The Rings',
    genre: 'fantasy',
    category: 'classic'
  }
}])

//Query Vector
const results = await namespace.query<Metadata>(
  {
    vector: [
      ... // query embedding
    ],
    includeVectors: true,
    includeMetadata: true
    topK: 1,
    filter: "genre = 'fantasy' and title = 'Lord of the Rings'"
  },
)

//Delete Record
await namespace.delete("upstash-rocks");

//Fetch records by their IDs
await namespace.fetch(["id-1", "id-2"]);

Metadata Filtering

If you wanna learn more about filtering check: Metadata Filtering

Troubleshooting

We have a Discord for common problems. If you can't find a solution, please open an issue.

Docs

See the documentation for details.

Contributing

Vector Database

Create a new index on Upstash and copy the url and token.

Running tests

bun run test

Building

bun run build

Contributing

Make sure you have Bun.js installed and have those relevant keys with specific vector dimensions:

## Vector dimension should be 3842
UPSTASH_VECTOR_REST_URL="XXXXX"
UPSTASH_VECTOR_REST_TOKEN="XXXXX"

vector-js's People

Contributors

aykutkardas avatar buggyhunter avatar cahidarda avatar dev-ups avatar enesakar avatar fahreddinozcan avatar joschan21 avatar mdogan avatar mdumandag avatar ogzhanolguncu avatar ytkimirti avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

vector-js's Issues

UpstashError: Forbidden: /upsert-data is not allowed

I'm receiving this error

UpstashError: Forbidden: /upsert-data is not allowed

when I'm trying to insert data into a vector database in my JavaScript code.

const result = await this.vectorIndex.upsert({id: "testId", data:"testData"});

this.vectorIndex is just an abstraction of new Index({ url, token })

This is the stack trace of the error

at f.request (/.../node_modules/.pnpm/@[email protected]/node_modules/@upstash/vector/dist/index.js:52:26)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at d.exec (/.../node_modules/.pnpm/@[email protected]/node_modules/@upstash/vector/dist/index.js:63:39)

Is there something wrong with my code or my settings in my upstash project?

Diffing context using a single index

I have an app where I ingest data for multiple users and I'm trying to use metadata to diff context per user/entity.
But I realized Upstash LangChain integration doesn't pass the filter (which I'm passing to the store retriever) to the query as the payload doesn't seem to have a metadata field. Is there a way achieve this? Thanks for your help.

Possibility to partition for multi-tenancy?

Hello,
I'm trying upstash-vector with langchain on a product for RAG conversation. Users are able to upload various kind of documents and they are able to query individual documents.
With other providers we used namespaces or partition keys, what is the equivalent here?

Thanks!

QueryMany type error

If single query list is given to queryMany like index.queryMany([{vector: [1, 0, 0], topK: 6}]), return value is one dimensional list just as normal query function. Return type of queryMany is defined as two dimensional list.

Mandatory vector field causes type errors

Upsert Command

vector-js/README.md

Lines 76 to 84 in 801c666

//Update Data
await index.upsert({
id: "upstash-rocks",
metadata: {
title: 'Star Wars',
genre: 'sci-fi',
category: 'classic'
}
});

Here it is shown upsert command can be called without the vector or data fields to update the metadata information in the index but this is not compatible with the defined type for argument of the upsert command in:
type VectorPayload<TMetadata> = {
id: number | string;
vector: number[];
metadata?: NoInfer<TMetadata>;
};
type DataPayload<TMetadata> = {
id: number | string;
data: string;
metadata?: NoInfer<TMetadata>;
};
type PayloadArray<TMetadata> = VectorPayload<TMetadata>[] | DataPayload<TMetadata>[];

constructor(
payload: VectorPayload<TMetadata> | DataPayload<TMetadata> | PayloadArray<TMetadata>
) {

Fetch, Range, Query

Commands fetch, query and range has a field called "includeVector" in their argument to specify whether vector field will be included in their return value. However their returned type specify vector field as always returned.

export type Vector<TMetadata = Record<string, unknown>> = {
id: string;
vector: number[];
metadata?: TMetadata;
};

type FetchCommandOptions = {
includeMetadata?: boolean;
includeVectors?: boolean;
};

export type FetchResult<TMetadata = Record<string, unknown>> = Vector<TMetadata> | null;

type RangeCommandPayload = {
cursor: number | string;
limit: number;
includeVectors?: boolean;
includeMetadata?: boolean;
};

export type RangeResult<TMetadata = Record<string, unknown>> = {
nextCursor: string;
vectors: Vector<TMetadata>[];
};

type QueryCommandPayload = {
topK: number;
filter?: string;
includeVectors?: boolean;
includeMetadata?: boolean;
} & ({ vector: number[]; data?: never } | { data: string; vector?: never });

export type QueryResult<TMetadata = Record<string, unknown>> = {
id: number | string;
score: number;
vector: number[];
metadata?: TMetadata;
};

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.