Intermittent playground crash about embedbase HOT 7 CLOSED

PS: need to improve error handling when using embedbase-js sdk at some level to determine

from embedbase.

need to try dot product instead of cos sim in pgvector:

...
  select
    STUFFHERE,
    (documents.embedding <#> embedding) * -1 as similarity
  from documents

  -- The dot product is negative because of a Postgres limitation, so we negate it
  and (documents.embedding <#> embedding) * -1 > match_threshold

  -- OpenAI embeddings are normalized to length 1, so
  -- cosine similarity and dot product will produce the same results.
  -- Using dot product which can be computed slightly faster.
  --
  -- For the different syntaxes, see https://github.com/pgvector/pgvector
  order by documents.embedding <#> embedding
  
  limit match_count;
end;
$$;

need to setup performance monitoring beforehand though

from embedbase.

pgvector/pgvector#82

from embedbase.

different thing that could be tried that will highly likely improve perf:

1. 
2. https://github.com/pgvector/pgvector#query-options
3. increase list size (because table starts growing beyond the optimal 100) CREATE INDEX ON items USING ivfflat (embedding vector_ip_ops) WITH (lists = 220);
4. use SCANN/FAISS + Supabase

from embedbase.

To update the index to use 220 lists, you'll need to first drop the existing index and then create a new index with the desired lists value. Here are the SQL commands to do that:

-- Drop the existing index
DROP INDEX documents_embedding_vector_cosine_ops_idx;

-- Create a new index with 220 lists
CREATE INDEX documents_embedding_vector_cosine_ops_idx
ON documents
USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 220);

Replace documents_embedding_vector_cosine_ops_idx with the actual name of your index if it's different.

Dropping and recreating an index can have some impact on your users, depending on your database's current usage and workload. Here's how it might affect your users:

1. Query performance: While the index is being dropped and recreated, any queries that rely on this index may experience slower performance because the database will need to do a full table scan instead of using the index.

2. Table lock: Depending on the PostgreSQL version and configuration, dropping and creating an index might lock the table or cause other queries to be blocked. This can cause delays for users trying to access the table during the index operation.

To minimize the impact on your users, consider performing the index update during a maintenance window or a period of low database usage. Additionally, you can use the CONCURRENTLY keyword when creating the new index to avoid locking the table:

-- Create a new index with 220 lists, concurrently
CREATE INDEX CONCURRENTLY documents_embedding_vector_cosine_ops_idx
ON documents
USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 220);

Note that you cannot use the CONCURRENTLY keyword when dropping an index. However, dropping an index is generally a quick operation and should not cause significant disruption.

from embedbase.

nvm all this. just need to distinct the select query when optimizing duplicates

from embedbase.

fixed 🚢🚢🚢🚢🚢

from embedbase.