Comments (3)
divine-taco
commented on August 10, 2024
I can reproduce this. BitsAndBytes works. But HQQ runs into memory issues even when setting context length very short (e.g. 512)
from fsdp_qlora.
KeremTurgutlu
commented on August 10, 2024
OOM seems to be happening in the model loading stage, we load and quantize pretrained weights in parallel, you can potentially manually set n_workers to a lower number here:
Line 622 in 0b57d37
from fsdp_qlora.
KeremTurgutlu
commented on August 10, 2024
cf61426 - should be able to load using HQQ without OOM on a 24GB GPU. Time to load increased from 5 mins to 10 mins. Will investigate further if this was caused by recent HQQ repo changes.
from fsdp_qlora.
Related Issues (20)
- bugs for fine-tune fsdp multinode HOT 1
- RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase HOT 3
- Fine tuning only runs on CPU HOT 4
- Adding NVMe SSDs to Enable and Accelerate 100B Model Fine-tuning on a Single GPU
- Example with AMD ROCm/HIP HOT 4
- Torch Compile?
- train.py script crashes when using HQQ HOT 1
- Bigger context size?
- /opt/conda/conda-bld/pytorch_1708025847130/work/aten/src/ATen/native/cuda/Loss.cu:250: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [14,0,0] Assertion `t >= 0 && t < n_classes` failed.
- Dual GPU training instantly powers off my desktop HOT 6
- Q on comparison with SFTTrainer
- Question about adding / training Mixtral HOT 1
- Why is o_proj not targetted?
- nan when the input length is large HOT 5
- how to inference using 70b? or we need to implement it with the same way to train it by ourself? HOT 1
- process 0 terminated with signal SIGKILL HOT 4
- How to load the saved model?
- What if I have three graphics cards? HOT 1
- Results after running
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fsdp_qlora.