https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/blob/main/codellama-7b.Q4_0.gguf
and put them into weights
folder
# on MacOS using Homebrew (https://brew.sh/)
brew install ffmpeg
python -m venv venv
source venv/bin/activate
CMAKE_ARGS="-DLLAMA_METAL=on" pip install -U llama-cpp-python --no-cache-dir
pip install -r requirements.txt
First start up will be the longest, as the whisper model must be downloaded. After that the startup takes approx 5-10 seconds.
To start the bot run:
python main.py
Several modules have been implemented:
llama.c
as the main LLMsay
macOS command as the main TTS (text-to-speech)whisper
as the realtime speech recognizer STT (speech-to-text)langchain
as the agent workflow with prompt template and chat history
These issues are unresolved for now:
- Interrupt the saying command (the model can't be interrupted)
- More context (llama2 prefix-match limit is being hit fast)
- Better model with more compute (for now quantized llama is a little bit hallucinative)