Training in mixed precision about i-bert HOT 3 CLOSED

I manually disabled autocasting in the linear blocks and got the forward pass to work, but I'm getting nans now in the backward pass.
I'll update with more details if I'm able to get the train the model in a stable manner in mixed precision.

from i-bert.

Hi, thanks for your interest and I apologize for my late response.
Do you still encounter the same problem?
Since we haven't tried with lower precision than the default 8-bit setting, we have not encountered the same issue.
It is probably because we haven't taken into account lower bit precisions when writing the code and there might be some corner cases we haven't debugged.
In case you have already had a solution for this, we would also greatly appreciate it if you could please open up a PR for it.

from i-bert.

Thanks for your response!

I got around that issue by disabling autocasting in the linear blocks but I realized after doing that, that it defeats the purpose of mixed precision training because most of the computations are in the linear layers which are in fp32 (after the disabling), so it yielded no improvement in training time.
I've given up on it for now because I had to move on to other things, but I'll definitely provide an update if I get it working in the future.

from i-bert.