iamskydra / finetune-quantize-llms Goto Github PK

This repository contains materials for the CSE Summer School Hackathon 2024, aimed at guiding high school students on how to fine-tune and quantize large language models (LLMs) with Google Colab in the simplest way possible. It minimizes the need for coding and requires only minor adjustments to pre-designed functions.

Key information:

• Fine-tuning the model with LLaMA Factory.
• Quantizing the model with Unsloth.
• The model being fine-tuned in the notebook is GemSUra 2B.
• The type of fine-tuning fixed in the notebook is LoRA using SFT with sample data from HCMUT_FAQ. Students are guided on how to create their own data based on the competition theme.
• The quantization method in the notebook is 8-bit, enabling students to deploy the model on personal computers using Ollama.

!pip install --upgrade -r requirements.txt -q

from module import *

hf("HF_TOKEN")

hcmut_dataset = load_dataset("IAmSkyDra/HCMUT_FAQ", split="train", streaming=False)

identity_dataset = load_dataset("IAmSkyDra/HCMUT_FAQ", split="validation", streaming=False)

train([identity_dataset, hcmut_dataset], num_train_epochs, continue_training)

test()

merge_and_push("IAmSkyDra/GemSUra-edu")

inference("IAmSkyDra/GemSUra-edu")

chat(max_new_tokens, history)

quantize_and_push("IAmSkyDra/GemSUra-edu-quantized")

Detailed explanations of each function and their arguments, as well as other information, are provided in the notebook.

By the end of this notebook, students will have a fine-tuned model tailored with custom data based on the pre-trained GemSUra 2B model, along with its quantized version.