Python script to demonstrate how to invoke models such as LLaMA from the command-line, with LoRA adapters.

Uses the huggyllama/llama-7b LLaMA distribution by default, but if you have the official LLaMA weights and would prefer to convert them to Huggingface format yourself: I provide instructions for doing so.
Uses the alpaca-lora-7b LoRA by default, to adapt LLaMA for instruction-following.

All instructions are written assuming your command-line shell is bash.

Clone repository:

git clone https://github.com/Birch-san/llama-play.git
cd llama-play

This is to avoid interfering with your current Python environment (other Python scripts on your computer might not appreciate it if you update a bunch of packages they were relying on).

Follow the instructions for virtualenv, or conda, or neither (if you don't care what happens to other Python scripts on your computer).

Create environment:

. ./venv/bin/activate
pip install --upgrade pip

Activate environment:

. ./venv/bin/activate

(First-time) update environment's pip:

pip install --upgrade pip

Download conda.

Skip this step if you already have conda.

Install conda:

Skip this step if you already have conda.

Assuming you're using a bash shell:

# Linux installs Anaconda via this shell script. Mac installs by running a .pkg installer.
bash Anaconda-latest-Linux-x86_64.sh
# this step probably works on both Linux and Mac.
eval "$(~/anaconda3/bin/conda shell.bash hook)"
conda config --set auto_activate_base false
conda init

Create environment:

conda create -n p311-llama python=3.11

Activate environment:

conda activate p311-llama

Ensure you have activated the environment you created above.

(Optional) treat yourself to latest nightly of PyTorch, with support for Python 3.11 and CUDA 12.1:

# CUDA
pip install --upgrade --pre torch --extra-index-url https://download.pytorch.org/whl/nightly/cu121

Install dependencies:

pip install -r requirements.txt

From root of repository:

python -m scripts.chat_play --bf16

This repository is itself MIT-licensed.

Includes MIT-licensed code copied from Artidoro Pagnoni's qlora, and Apache-licensed code copied from MosaicML's mpt-7b-chat Huggingface Space.