QLoRA with AutoGPTQ for quantization
I release the resources associated with GPTQLoRA finetuning in this repository under MIT license.
conda create -n gptqlora python=3.8
conda activate gptqlora
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117
git clone -b peft_integration https://github.com/PanQiWei/AutoGPTQ.git && cd AutoGPTQ
pip install .[triton]
cd ..
git clone https://github.com/timdettmers/bitsandbytes.git
cd bitsandbytes
# CUDA_VERSIONS in {110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 120}
# make argument in {cuda110, cuda11x, cuda12x}
# if you do not know what CUDA you have, try looking at the output of: python -m bitsandbytes
CUDA_VERSION=117 make cuda11x
python setup.py install
cd ..
pip install git+https://github.com/huggingface/transformers.git
pip install git+https://github.com/huggingface/peft.git
pip install git+https://github.com/huggingface/accelerate.git
pip install -r requirements.txt
pip install protobuf==3.20.*The gptqlora.py code is a starting point for finetuning and inference on various datasets.
Basic command for finetuning a baseline model on the Alpaca dataset:
python gptqlora.py --model_path <path>For models larger than 13B, we recommend adjusting the learning rate:
python gptqlora.py –learning_rate 0.0001 --model_path <path>The file structure of the model checkpoint is as follows:
(bnb) root@/root/qlora-main# ls llama-7b/
config.json gptq_model-4bit-128g.bin special_tokens_map.json tokenizer_config.json
generation_config.json quantize_config.json tokenizer.model
Quantization is based on AutoGPTQ. Also, to run the code, you first need a model converted to GPTQ.
You can access the paged optimizer with the argument --optim paged_adamw_32bit
This code is based on QLoRA.
This repo builds on the Stanford Alpaca and LMSYS FastChat repos.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent