batch_size > 1 results in NaN loss value about ludwig HOT 11 OPEN

Hi @K-Mistele! This is actually a known issue that we recently debugged and is actually not specific to Ludwig!

The best way to solve it is to set bnb_4bit_compute_dtype in the quantisation section of the Ludwig config to bfloat16 instead of float16 since batch sizes of > 1 with mistral in particular lead to bit overflows during training resulting in NaN loss during the first backprop in the train loop.

However, I notice you're training on a V100 and I don't think bfloat16 is supported since it only works on ampere architectures and above? Is there any chance you can use a newer Nvidia GPU?

from ludwig.

The only Nvidia GPU that supports the bfloat16 is the A100 which I do not have access to. My v100 is an owned GPU not a rented/cloud one, so I try and stick with that whenever possible since I'm not paying by the hour.

from ludwig.

@K-Mistele that makes sense! Actually the entire A series uses Ampere, so you could consider an A5000 from AWS which is pretty cheap. I might also suggest giving the Predibase free trial a try since we have A5000s/A6000s etc (A10Gs) for fine-tuning and we have $25 in free trial credits!

from ludwig.

I am planning to I just want to make sure I can use the tool locally first

from ludwig.

is there no workaround for a v100?

from ludwig.

Unfortunately, not to my knowledge with Mistral. Do you want to test Llama-2-7B instead? The issue doesn't show up there with larger batch sizes!

from ludwig.

yeah I can try it

from ludwig.

@K-Mistele let me know how it goes!

from ludwig.

Do you know if zephyr has the same problem @arnavgarg1 ?

from ludwig.

@K-Mistele not to my knowledge!

from ludwig.

@K-Mistele Did the fix work?

from ludwig.