different-ai / embedbase Goto Github PK

A dead-simple API to build LLM-powered apps

License: MIT License

Dockerfile 0.06% Makefile 1.08% Python 31.72% Shell 0.02% PLpgSQL 0.35% TypeScript 59.09% JavaScript 0.81% CSS 0.25% Solidity 1.13% Jupyter Notebook 5.39% HTML 0.09%

ai artificial-intelligence embeddings machine-learning natural-language-processing openai pgvector supabase chatgpt chatgpt-plugin gpt-4 vector-database

embedbase's Introduction

Embedbase

All the tools you need to develop AI apps

Try the Hosted Version · Request Feature · Report Bug

Check out the docs for more info.

What is it

Embedbase is a dead-simple API to help you use VectorDBs and LLMs without needing to host them!

Key features

Generate: use .generateText() to use 9+ LLMs
Semantic Search: use .add() to create a list of semantically searchable information and .search() to run semantic queries

Installation

npm i embedbase-js

import { createClient } from 'embedbase-js'
// initialize client
const embedbase = createClient(
  'https://api.embedbase.xyz',
  '<grab me here https://app.embedbase.xyz/>'
)
 
const question =
  'im looking for a nice pant that is comfortable and i can both use for work and for climbing'
 
// search for information in a pre-defined dataset and returns the most relevant data
const searchResults = await embedbase.dataset('product-ads').search(question)
 
// transform the results into a string so they can be easily used inside a prompt
const stringifiedSearchResults = searchResults
  .map(result => result.data)
  .join('')
 
const answer = await embedbase
  .useModel('openai/gpt-3.5-turbo')
  .generateText(`${stringifiedSearchResults} ${question}`)
 
console.log(answer) // 'I suggest considering harem pants for your needs. Harem pants are known for their ...'

Getting started
Javascript SDK
Docs and support
Integrations
Contributing

What are people building

The fastest way to get started with Embedbase is signing up for free to Embedbase Cloud.

Docs and support

Check out our tutorials for step-by-step guides, how-to's, and best practices, our documentation is powered by GPT-4, so you can ask question directly.

Ask a question in our Discord community to get support.

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

embedbase's People

Contributors

Stargazers

Watchers

embedbase's Issues

Rename "search" directory to "embedbase"

ValueError: Out of range float values are not JSON compliant

embedbase/embedbase/app.py

Line 159 in d494ec0

return JSONResponse(

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 436, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 333, in _sentry_patched_asgi_app
    return await middleware(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py", line 139, in _run_asgi3
    return await self._run_app(scope, lambda: self.app(scope, receive, send))
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py", line 188, in _run_app
    raise exc from None
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py", line 183, in _run_app
    return await callback()
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 130, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 130, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 130, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 108, in __call__
    response = await self.dispatch_func(request, call_next)
  File "/app/middlewares/auth_api_key/auth_api_key.py", line 182, in dispatch
    response = await call_next(request)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 84, in call_next
    raise app_exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 70, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 227, in _sentry_exceptionmiddleware_call
    await old_call(self, scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 130, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 130, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/fastapi.py", line 130, in _sentry_app
    return await old_app(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "/app/embedbase/app.py", line 159, in add
    return JSONResponse(
  File "/usr/local/lib/python3.10/site-packages/starlette/responses.py", line 196, in __init__
    super().__init__(content, status_code, headers, media_type, background)
  File "/usr/local/lib/python3.10/site-packages/starlette/responses.py", line 55, in __init__
    self.body = self.render(content)
  File "/usr/local/lib/python3.10/site-packages/starlette/responses.py", line 199, in render
    return json.dumps(
  File "/usr/local/lib/python3.10/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/local/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
ValueError: Out of range float values are not JSON compliant

[Core/Hosted]: search feedback

Feature request

ChatGPT like user feedback to open opportunities to improve user experience

Motivation

interpretability over user happiness as statistics, visualisation in the dashboard
eventual fine-tuning or reranking

Your contribution

I'm imagining a middleware that save search queries and a new end point that accept feedback on search result with the ID of the result

from typing import Callable

import uuid

from fastapi import Request
from fastapi.responses import JSONResponse

from embedbase.database.base import VectorDatabase
from embedbase.embedding.base import Embedder


async def save_search(
    request: Request, call_next: Callable, db: VectorDatabase, embedder: Embedder
):
    """
    Upon search request, save the request to a database.
    """
    # todo overlap with add on "search" dataset
    if request.method != "POST" or "/v1/search" not in request.url.path:
        return await call_next(request)

    request_body = await request.json()
    new_id = str(uuid.uuid4())
    request_body["id"] = new_id
    response = await db.save("search", request_body)
    return await call_next(request)

(almost) pseudo code for feedback endpoint:

app = (
    get_app()
    .use_embedder(...)
    .use_db(...)
    .run()
)
# An endpoint that let you rate search results
@app.post("/feedback")
async def human_feedback(req, cb, db, embedder):
    # here would save to a table feedback
    # the request body looks like "searchid: vrevrwrew, feedback: 0 or 1"
    db.save("feedback", req.body)
    return 200

"Show links" doesn't work

First of all, that functionality looks very promising.

I updated the plugin, logged out and in, restarted, cleared cache, disabled enabled... but the plugin doesn't work. I don't know if I have to wait for a long time...

async function call not working (uvicorn, fastapi)

🐛 Bug Report

async function call not working (uvicorn, fastapi)

Code sample

from embedbase_client.client import EmbedbaseClient
import yaml
from fastapi import FastAPI

with open("credential/embedbase.yaml", "r") as f:
    API = yaml.safe_load(f)
embedbase_key = API['EMBEDBASE_KEY']
embedbase_url = API['EMBEDBASE_URL']


app = FastAPI()


@app.get("/")
async def root():
    client = EmbedbaseClient(embedbase_url, embedbase_key)

    client.dataset('basil_greenhouse').search(
        'how to grow basil?', limit=5)
    return {"message": "Hello World"}

uvicorn bug_test:app

Environment

OS: Linux
Python version, get it with: 3.8.16

Screenshots

RuntimeWarning: Enable tracemalloc to get the object allocation traceback

📈 Expected behavior

Return list of datasets

📎 Additional context

Using FastAPI

Integrate seamlessly with GPT Index (aka llamaindex)

Publish Docker image without authentication

i.e. multiple tags
Especially firebase_admin seems a huge dependency

https://github.com/another-ai/embedbase/blob/main/docker/Dockerfile#L12

Advanced duplicate filtering using heuristics

see https://community.openai.com/t/can-this-api-be-used-to-query-internal-data/122091/28?u=louis030195

UserID should be handled on the implementer side

https://github.com/different-ai/embedbase/blob/cf3b6952bf3be4631952b2c16604141f375ed4dd/embedbase/app.py#LL308

Custom /_error page without a custom /404 page

Hi,

I have tried to to npm run dev the dashboard part yet it shows
You have added a custom /_error page without a custom /404 page. This prevents the 404 page from being auto statically optimized. See here for info: https://nextjs.org/docs/messages/custom-error-no-custom-404

fix ci unit test setup

Automate Supabase configuration

https://github.com/different-ai/embedbase-docs/blob/main/pages/self-hosting/deployment.mdx

[Core]: ValueError: Out of range float values are not JSON compliant

System Info

hosted

Reproduction

add with empty data maybe?

Expected behavior

ValueError: Out of range float values are not JSON compliant

Implement Weaviate

weaviate.io

invalid json shouldn't 500 but rather 400

How to use with local embedbase deployment?

https://docs.embedbase.xyz/tutorials/nextra-qa-docs is only setup for a cloudhosted embedbase - is there a way to get this to run with a local deployment?

Metadata filtering

De-correlate vector storage / engine & general storage

I believe we should store only vectors in vector databases and everything else in another storage (firestore, polybase, mongo, postgres, whatever) (+ eventually the vectors)

https://app.excalidraw.com/l/5MKXLddifTr/2Z7ROMvo9Yg

[Python SDK, Javascript SDK]: improve error handling

Feature request

When something is wrong it should show clearly the problem to the user
Unit tested using mock for example

Motivation

Your contribution

How to use api createContext

I’m using your embedbase library on python. I wonder how can I use the createContext api like on the embedbase.xyz playground menu. By the way embedbasd is very helpful to me. Thank you very much.

Add Supabase Auth

Getting started examples

🚀 Feature Request

colab
replit - done but need to document
code sandbox - done but need to document
etc

🔈 Motivation

Getting started quickly without having to install anything locally (using in memory db, no docker stuff)

Local computation and storage

Local model and storage for data privacy concerned people

Multiple paths to embed large documents

https://app.excalidraw.com/l/5MKXLddifTr/458MijLwNCU

[Dashboard, Core, Python SDK, Javascript SDK]: integration unit tests concurrency issue

System Info

Reproduction

run local test in the same time than ci or multiple projects integration tests running in parallel

Expected behavior

pass

run tests on different datasets according to environment/context

[Core]: tiktoken stackoverflow

System Info

Reproduction

send 1000000000000000000000000 length string to add

pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: StackOverflow

at .encode ( [/usr/local/lib/python3.10/site-packages/tiktoken/core.py:120](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Ftiktoken%2Fcore.py&line=120&project=embedbase) )
at .is_too_big ( [/usr/local/lib/python3.10/site-packages/embedbase/embedding/openai.py:72](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fembedbase%2Fembedding%2Fopenai.py&line=72&project=embedbase) )
at .add ( [/usr/local/lib/python3.10/site-packages/embedbase/app.py:140](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fembedbase%2Fapp.py&line=140&project=embedbase) )
at .run_endpoint_function ( [/usr/local/lib/python3.10/site-packages/fastapi/routing.py:163](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Ffastapi%2Frouting.py&line=163&project=embedbase) )
at .app ( [/usr/local/lib/python3.10/site-packages/fastapi/routing.py:237](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Ffastapi%2Frouting.py&line=237&project=embedbase) )
at ._sentry_app ( [/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/fastapi.py:130](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fsentry_sdk%2Fintegrations%2Ffastapi.py&line=130&project=embedbase) )
at .app ( [/usr/local/lib/python3.10/site-packages/starlette/routing.py:66](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fstarlette%2Frouting.py&line=66&project=embedbase) )
at .handle ( [/usr/local/lib/python3.10/site-packages/starlette/routing.py:276](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fstarlette%2Frouting.py&line=276&project=embedbase) )
at .__call__ ( [/usr/local/lib/python3.10/site-packages/starlette/routing.py:718](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fstarlette%2Frouting.py&line=718&project=embedbase) )
at .__call__ ( [/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py:18](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Ffastapi%2Fmiddleware%2Fasyncexitstack.py&line=18&project=embedbase) )
at ._create_span_call ( [/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py:130](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fsentry_sdk%2Fintegrations%2Fstarlette.py&line=130&project=embedbase) )
at .__call__ ( [/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py:68](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fstarlette%2Fmiddleware%2Fexceptions.py&line=68&project=embedbase) )
at ._create_span_call ( [/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py:130](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fsentry_sdk%2Fintegrations%2Fstarlette.py&line=130&project=embedbase) )
at ._sentry_exceptionmiddleware_call ( [/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py:229](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fsentry_sdk%2Fintegrations%2Fstarlette.py&line=229&project=embedbase) )
at .coro ( [/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py:70](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fstarlette%2Fmiddleware%2Fbase.py&line=70&project=embedbase) )
at .__aexit__ ( [/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py:662](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fanyio%2F_backends%2F_asyncio.py&line=662&project=embedbase) )
at .__call__ ( [/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py:106](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fstarlette%2Fmiddleware%2Fbase.py&line=106&project=embedbase) )
at ._create_span_call ( [/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py:130](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fsentry_sdk%2Fintegrations%2Fstarlette.py&line=130&project=embedbase) )
at .__call__ ( [/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py:83](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fstarlette%2Fmiddleware%2Fcors.py&line=83&project=embedbase) )
at ._create_span_call ( [/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py:130](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fsentry_sdk%2Fintegrations%2Fstarlette.py&line=130&project=embedbase) )
at .__call__ ( [/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py:162](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fstarlette%2Fmiddleware%2Ferrors.py&line=162&project=embedbase) )
at ._create_span_call ( [/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py:130](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fsentry_sdk%2Fintegrations%2Fstarlette.py&line=130&project=embedbase) )
at .__call__ ( [/usr/local/lib/python3.10/site-packages/starlette/applications.py:122](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fstarlette%2Fapplications.py&line=122&project=embedbase) )
at ._run_app ( [/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py:183](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fsentry_sdk%2Fintegrations%2Fasgi.py&line=183&project=embedbase) )
at ._run_asgi3 ( [/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py:139](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fsentry_sdk%2Fintegrations%2Fasgi.py&line=139&project=embedbase) )
at ._sentry_patched_asgi_app ( [/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py:335](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fsentry_sdk%2Fintegrations%2Fstarlette.py&line=335&project=embedbase) )
at .__call__ ( [/usr/local/lib/python3.10/site-packages/fastapi/applications.py:276](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Ffastapi%2Fapplications.py&line=276&project=embedbase) )
at .__call__ ( [/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py:78](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fuvicorn%2Fmiddleware%2Fproxy_headers.py&line=78&project=embedbase) )
at .run_asgi ( [/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py:435](https://console.cloud.google.com/debug?referrer=fromlog&file=%2Fusr%2Flocal%2Flib%2Fpython3.10%2Fsite-packages%2Fuvicorn%2Fprotocols%2Fhttp%2Fhttptools_impl.py&line=435&project=embedbase) )

Expected behavior

simple fix: if len(s) > 10000 > then check tokens (no need to count tokens if the string is incredibly long already)

Implement online vs batching

Support for local databases

Great project. Would be nice to get support for a local DB, without having to rely on a third party. Maybe a local PostgreSQL or local SQLite?

Ignore empty inputs

Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/tenacity/init.py", line 382, in call result = fn(*args, **kwargs) File "/app/embedbase/api.py", line 197, in embed return openai.Embedding.create(input=input, model=model)["data"] File "/usr/local/lib/python3.10/site-packages/openai/api_resources/embedding.py", line 33, in create response = super().create(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 153, in create response, _, api_key = requestor.request( File "/usr/local/lib/python3.10/site-packages/openai/api_requestor.py", line 226, in request resp, got_stream = self._interpret_response(result, stream) File "/usr/local/lib/python3.10/site-packages/openai/api_requestor.py", line 619, in _interpret_response self._interpret_response_line( File "/usr/local/lib/python3.10/site-packages/openai/api_requestor.py", line 679, in _interpret_response_line raise self.handle_error_response( openai.error.InvalidRequestError: [''] is not valid under any of the given schemas - 'input'

[Python SDK]: async combining two dataset doesn't work after python SDK update

System Info

embedbase-client: 0.1.2
ubuntu 22.04
Python 3.8.16

Reproduction

client = EmbedbaseAsyncClient(embedbase_url, embedbase_key)
if farm_id is None:
    # this works!
    [results] = await asyncio.gather(
        *[client.dataset(recipe_id).search(question, limit=7).get()]
    ) 
elif recipe_id is None:
    # this does not work!

    recipe_id = getRecipeId(farm_id)
    [results_recipe, results_farm] = await asyncio.gather(
        *[
            client.dataset(recipe_id).search(question, limit=6).get(),
            client.dataset(farm_id).search(question, limit=1).get(),
        ]
    )

Expected behavior

two dataset combined and return

Add Auth0 support

Add references to github and pdf uploader

🚀 Feature Request

get footnotes ref with github and pdf uploader in playground or search responses

🔈 Motivation

🛰 Alternatives

📎 Additional context

copy pasta docs script code or other stuff just throw urls in metadata path and at generation time include metadata

Intermittent playground crash

🐛 Bug Report

crash sometimes (chain of issues, backend crashing default timeout supabase is 5 seconds)

🔬 How To Reproduce

Steps to reproduce the behavior:

Ask a question that involves a big context I guess (example: ask a question to whole ethereum documentation)

📎 Additional context

Dashboard/playground

TypeError: Cannot read properties of undefined (reading 'data')
  File "app:///_next/server/pages/api/createContext.js", line 139, col 18, in <anonymous>
    return r.data;
  File "<anonymous>", in Array.map
  File "app:///_next/server/pages/api/createContext.js", line 132, col 30, in createContext
    const datas = topResults.map((r)=>{
  File "node:internal/process/task_queues", line 95, col 5, in process.processTicksAndRejections
  File "app:///_next/server/pages/api/createContext.js", line 170, col 21, in buildPrompt
    const context = await createContext(prompt, datasetIds, apiKey);
  File "/var/task/dashboard/node_modules/@sentry/nextjs/cjs/server/wrapApiHandlerWithSentry.js", line 143, col 33, in <anonymous>
    const handlerResult = await wrappingTarget.apply(thisArg, args);
  File "/var/task/dashboard/node_modules/next/dist/server/api-utils/node.js", line 372, col 9, in Object.apiResolver
    await resolver(req, res);
  File "/var/task/dashboard/node_modules/next/dist/server/next-server.js", line 513, col 9, in NextNodeServer.runApi
    await (0, _node1).apiResolver(req.originalRequest, res.originalResponse, query, pageModule, {
  File "/var/task/dashboard/node_modules/next/dist/server/next-server.js", line 815, col 35, in Object.fn
    handled = await this.handleApiRequest(req, res, query, // TODO: see if we can add a runtime check for this
  File "/var/task/dashboard/node_modules/next/dist/server/router.js", line 243, col 32, in Router.execute
    const result = await route.fn(req, res, params, parsedUrlUpdated, upgradeHead);

Backend

different-ai/embedbase-hosted#3

Solutions:

increase timeout supabase-community/supabase-py#376
SQL optimisation (better)

Clarify and write down philosophy/direction/opinions

[Dashboard]: chat kept in state

System Info

Reproduction

chat on the dashboard
go to another tab
come back

messages stay here (need to clear react state)

Expected behavior

new chat everytime

merge docs apis into a single one using wasm hacks

https://github.com/different-ai/embedbase/blob/main/docs/pages/api/ask.ts

Search over multiple datasets

Use Alpine in Docker

[Core]: add response model

Feature request

add response model to endpoints

Motivation

hard to parse

Your contribution

[Dashboard, Core, Python SDK, Javascript SDK]: publish appropriate changelog on release

Feature request

feat

fix

chore

only from the matching project, example:

Bob commit "feat(dashboard): add gpt5"

John commit "fix(hosted): fix bug"

Alice commit "chore(js): add something"

Releases

js-1.0.1

🧹 Chore 🧹

Alice commit "chore(js): add something"

dashboard-1.6.7

⭐️ Features ⭐️

Bob commit "feat(dashboard): add gpt5"

etc

Motivation

Your contribution

Fail to show data table

In embedbase cloud, to view the data table I am supposed to go to this url (dev as my datasets name), https://app.embedbase.xyz/dashboard/explorer/dev, however it fails to show the data.

I need to click the arrow and lead to this https://app.embedbase.xyz/dashboard/explorer/dev?page=0 to show the data table. In addition, further clicking of left arrow will lead to
https://app.embedbase.xyz/dashboard/explorer/dev?page=-1
https://app.embedbase.xyz/dashboard/explorer/dev?page=-2

Create issue & fr templates

Implement automatic top k

Local embeder also use openai dimensions

When follow https://docs.embedbase.xyz/examples/local-embedder. I got error below.

INFO: 127.0.0.1:58500 - "POST /v1/animals HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 436, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/usr/local/lib/python3.8/dist-packages/uvicorn/middleware/proxy_headers.py", line 78, in call
return await self.app(scope, receive, send)
File "/usr/local/lib/python3.8/dist-packages/fastapi/applications.py", line 276, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.8/dist-packages/starlette/applications.py", line 122, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/exceptions.py", line 79, in call
raise exc
File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/exceptions.py", line 68, in call
await self.app(scope, receive, sender)
File "/usr/local/lib/python3.8/dist-packages/fastapi/middleware/asyncexitstack.py", line 21, in call
raise e
File "/usr/local/lib/python3.8/dist-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
await self.app(scope, receive, send)
File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 718, in call
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 66, in app
response = await func(request)
File "/usr/local/lib/python3.8/dist-packages/fastapi/routing.py", line 237, in app
raw_response = await run_endpoint_function(
File "/usr/local/lib/python3.8/dist-packages/fastapi/routing.py", line 163, in run_endpoint_function
return await dependant.call(**values)
File "/usr/local/lib/python3.8/dist-packages/embedbase/app.py", line 230, in add
await self.db.update(
File "/usr/local/lib/python3.8/dist-packages/embedbase/database/postgres_db.py", line 197, in update
cur.execute(q, flat_values)
File "/usr/local/lib/python3.8/dist-packages/psycopg/cursor.py", line 723, in execute
raise ex.with_traceback(None)
psycopg.errors.DataException: expected 1536 dimensions, not 384

For gpt3 connections

Add .ratio to embedbase sdk

It would be so helpful in my use-case

results = await client.dataset(recipe_id, farm_id, user_id, locaton_id).search(question, max_token=3000, ratio=[.7, .1, .1, .1)

max_token of 2100, 300, 300, 300 are applied to each dataset.

Originally posted by @ccomkhj in #71 (comment)

Public datasets

metadata unicode issue with supabase

🐛 Bug Report

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 375, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 333, in _sentry_patched_asgi_app
    return await middleware(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py", line 139, in _run_asgi3
    return await self._run_app(scope, lambda: self.app(scope, receive, send))
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py", line 188, in _run_app
    raise exc from None
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py", line 183, in _run_app
    return await callback()
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 130, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 130, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 130, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 108, in __call__
    response = await self.dispatch_func(request, call_next)
  File "/middlewares/auth_api_key/auth_api_key.py", line 182, in dispatch
    response = await call_next(request)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 84, in call_next
    raise app_exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 70, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 227, in _sentry_exceptionmiddleware_call
    await old_call(self, scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 130, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 130, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/fastapi.py", line 130, in _sentry_app
    return await old_app(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/local/lib/python3.10/site-packages/embedbase/app.py", line 213, in add
    await self.db.update(
  File "/usr/local/lib/python3.10/site-packages/embedbase/database/supabase_db.py", line 125, in update
    await asyncio.gather(*[_insert(batch_df) for batch_df in batches])
  File "/usr/local/lib/python3.10/site-packages/embedbase/database/supabase_db.py", line 122, in _insert
    .execute()
  File "/usr/local/lib/python3.10/site-packages/postgrest/_sync/request_builder.py", line 68, in execute
    raise APIError(r.json())
postgrest.exceptions.APIError: {'code': '22P05', 'details': '\\u0000 cannot be converted to text.', 'hint': None, 'message': 'unsupported Unicode escape sequence'}

🔬 How To Reproduce

Steps to reproduce the behavior:

Code sample

Environment

OS: [e.g. Linux / Windows / macOS]
Python version, get it with:

python --version

Screenshots

📈 Expected behavior

📎 Additional context

Add code snippets in JS to show how to use api

I find it a bit hard to parse curl requests and I think the readme would benefit from an example with fetch

Add examples

Would be cool to see a few extra examples.

How would embedbase work in various use case. Here's a couple cool repos for inspo:
https://github.com/dandv/shower-thoughts-weaviate
https://github.com/gil--/shoppy-gpt
https://github.com/gmchad/embed-vtt

different-ai / embedbase Goto Github PK

embedbase's Introduction

Embedbase

All the tools you need to develop AI apps

What is it

Key features

Installation

Table of Contents

What are people building

Docs and support

Contributing

embedbase's People

Contributors

Stargazers

Watchers

Forkers

embedbase's Issues

Feature request

Motivation

Your contribution

🐛 Bug Report

Code sample

Environment

Screenshots

📈 Expected behavior

📎 Additional context

System Info

Reproduction

Expected behavior

Feature request

Motivation

Your contribution

🚀 Feature Request

🔈 Motivation

System Info

Reproduction

Expected behavior

System Info

Reproduction

Expected behavior

System Info

Reproduction

Expected behavior

🚀 Feature Request

🔈 Motivation

🛰 Alternatives

📎 Additional context

🐛 Bug Report

🔬 How To Reproduce

📎 Additional context

System Info

Reproduction

Expected behavior

Feature request

Motivation

Your contribution

Feature request

Releases

🧹 Chore 🧹

⭐️ Features ⭐️

Motivation

Your contribution

🐛 Bug Report

🔬 How To Reproduce

Code sample

Environment

Screenshots

📈 Expected behavior

📎 Additional context

Recommend Projects

Recommend Topics

Recommend Org