How do I run the Ollama [Bug]: about quivr HOT 4 OPEN

Hey @FaizanMunsaf! I'm here to help you with the Ollama issue you're facing. I'm a bot designed to assist with bugs, questions, and contributions. Let's work on resolving this together.

To run the Ollama private model using the current repository, follow these steps:

1. Install Ollama:
   Install Ollama from their website.

2. Run Ollama in the background:
   Execute the following command to run Ollama:

   ollama run llama2

3. Run Migrations:
   Move the migration file and reset the database:

   mv supabase/migrations/local_20240107152745_ollama.sql supabase/migrations/20240107152745_ollama.sql
   supabase db reset

4. Add Ollama Model to Quivr:
   Update the user_settings table in Supabase to include the Ollama models:

   [
     "ollama/llama2",
     "ollama/mistral"
   ]

   If you want to add it to your user only, you can add it to the models column in the user_settings table. If you want to set it as default for all new users, drop the entire table with the following command:

   DELETE TABLE user_settings;

5. Configure .env File:
   Update your .env file to activate the Ollama API by uncommenting and setting the OLLAMA_API_BASE_URL:

   # LOCAL
   OLLAMA_API_BASE_URL=http://host.docker.internal:11434

6. Integrate with Quivr:
   Ensure that the QuivrRAG class in quivr_rag.py is properly configured to use the Ollama model. The ChatLiteLLM class should be instantiated with the api_base parameter set to the OLLAMA_API_BASE_URL if the model starts with "ollama". This is handled in the _create_llm method of the QuivrRAG class:

   def _create_llm(
       self,
       callbacks,
       model,
       streaming=False,
       temperature=0,
   ) -> BaseLLM:
       if streaming and callbacks is None:
           raise ValueError(
               "Callbacks must be provided when using streaming language models"
           )
   
       api_base = None
       if self.brain_settings.ollama_api_base_url and model.startswith("ollama"):
           api_base = self.brain_settings.ollama_api_base_url
   
       return ChatLiteLLM(
           temperature=temperature,
           max_tokens=self.max_tokens,
           model=model,
           streaming=streaming,
           verbose=False,
           callbacks=callbacks,
           api_base=api_base,
       )

By following these steps, you will have the Ollama model running and integrated with Quivr ^[1][2][3].

To continue talking to Dosu, mention @dosu.

from quivr.

TECH-61 How do I run the Ollama [Bug]:

from quivr.

real problem is getting it to work with llama3. how do we do that?

from quivr.

how to run with llama3

from quivr.