Giter Club home page Giter Club logo

Comments (5)

puj avatar puj commented on May 27, 2024 4

Workaround: Re-index your data

Slack conversation

@dudanogueira Mentions that re-indexing your data should help. This does help. Although it's unclear what problem it actually solves. The linked instructions for migration are here

Caveat

You will need to create a separate collection with a different name. In my case, I will switch between OpenAI_MenuItemEmbeddings and OpenAI_MenuItemEmbeddings_SWAP each time I need to re-index.

Code for re-indexing

I did something like this

SOURCE_COLLECTION_NAME = "OpenAI_MenuItemEmbeddings"
CURRENT_COLLECTION_NAME = "OpenAI_MenuItemEmbeddings_SWAP"

# Grab the collection
menu_item_embeddings_collection = client.collections.get(CURRENT_COLLECTION_NAME)
menu_item_embeddings_collection_source = client.collections.get(SOURCE_COLLECTION_NAME)


# Migrate function to copy data from source to current
# We will use this when the vector cache becomes corrupt
def migrate_data(collection_src: Collection, collection_tgt: Collection):
    response = collection_src.aggregate.over_all(total_count=True)
    print("Found {} objects in source collection".format(response.total_count))
    with collection_tgt.batch.fixed_size(batch_size=100) as batch:
        for q in tqdm(collection_src.iterator(include_vector=True)):
            batch.add_object(
                properties=q.properties,
                vector=q.vector,
                uuid=q.uuid
            )

    return True

# We don't have a reliable way to check if the vector cache is corrupt, so we will comment/uncomment this line as needed
# migrate_data(menu_item_embeddings_collection_source, menu_item_embeddings_collection)

In my case, I just use CURRENT_COLLECTION_NAME throughout my program (regardless of the actual name value)

Multiple vectors

If you followed the documentation and are using multiple named vectors, you should change

-vector=q.vector["default"],
+vector=q.vector,

Alternatives

Another solution (to avoid the collection rename) could be to re-index twice.

  • Create collection OpenAI_MenuItemEmbeddings_SWAP and run migrate_data
  • Delete OpenAI_MenuItemEmbeddings - ⚠️Careful. Could be good to make a backup of your /var/weaviate folder
  • Create collection OpenAI_MenuItemEmbeddings and run migrate_data

You should now have a searchable OpenAI_MenuItemEmbeddings.

from weaviate.

objectkuan avatar objectkuan commented on May 27, 2024 1

Getting the same error making our product unusable. Is there any workaround before a fix landed?

from weaviate.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.