What the recommended GPU setup for fine-tuning ? about yarn HOT 5 CLOSED

Could you please clarify if this discussion is around full parameter tuning or lora based? @bloc97

from yarn.

Thanks a lot. To confirm, A100 is with 40G or 80G memory for 7b 64k fine-tuning?

from yarn.

We were able to train the 7b 64k model on an 8x A100 node -- all other models unfortunately require a multinode setup. We used 64 GPUs, but I expect 16 would suffice for all other models (7b 128k, 13b 64k, 13b 128k)

from yarn.

It is 8x80GB for 64k context size

from yarn.

We were able to train the 7b 64k model on an 8x A100 node -- all other models unfortunately require a multinode setup. We used 64 GPUs, but I expect 16 would suffice for all other models (7b 128k, 13b 64k, 13b 128k)

I ran finetune.py using 2x A100 GPUs, and both GPUs loaded up to 14g/80g. After processing the first batch, the memory usage went up to 77g/80g, and then it ran OOM when starting the second batch.
Is this situation normal?

from yarn.