OOM Error: CUDA out of memory when finetuning llama3-8b about litgpt HOT 3 CLOSED

Hm, it could be related to the slightly larger size.

llama2-7b by the lit-llama

I think llama 2 is not supported by lit-llama. Do you perhaps meant llama 7B in lit-llama or llama 2 7B in LitGPT?

If you meant lit-llama, I am curious, does the 7B Llama 2 model work for you in LitGPT?

In any case, you could perhaps try QLoRA or a smaller sequence length to make it work.

from litgpt.

With --quantize bnb.nf4, I am able to fine-tune the Llama 3-8B without any problem on a single A10 GPU.

from litgpt.

Thanks for all the help. I found that the OOM error vanishes when I choose a smaller max-seq-length. I believe it's because my dataset samples are too long, leading to OOM. When I tried Lora with the Alpaca-2k dataset, it consumed 20.5GB of memory. When I used my dataset without limiting max_seq_length, it would OOM regardless of whether I used --quantize bnb.nf4 or not. The issue was resolved when I limited --max-seq-length 512.

from litgpt.