discuss the solutions to Not fully recovering spikes about ml-engineering HOT 7 CLOSED

Am I understanding that you'd like to keep this issue open for some time and invite a community discussion, correct?

Usually in this kind of situation we would roll back to a checkpoint right before the spike (but some 10-100 steps back) and then either try a new data (if bad data was the trigger) or lower the lr. And observe the grad norm while you're at it, since most likely your grads are spikes.

Publicly available training LLM/VLM logbooks is a good source of learning of how such problems were dealt with by others.

from ml-engineering.

yeah! Or close it when we have systematic solutions/ debugging guidances to this problem. I feel like the issue section is a great place for ppl to talk about their experiences dealing with model training at scale.

Thanks for the guidance! Will share my experiences later :)

from ml-engineering.

Thanks!! @stas00

from ml-engineering.

My solutions are:

1. increase the batch size
2. reduce the lr

However, I think the problem is somewhere else. I am sampling training data based on sequence length, i.e., a batch of sequences with similar lengths are sampled together. Thus the reason why the loss spikes is it switches to train on longer sequence batches.

from ml-engineering.

Hi @stas00 A random question. What is your principle for tuning the learning rate? The larger the better? My intuition is using the largest lr to make the learning phrase converge faster.

from ml-engineering.

@pengzhangzhi, it is probably better if we use the github Discussion feature for what you intend and leave Issues for what they are.

I have just enabled it here https://github.com/stas00/ml-engineering/discussions - so you can start a discussion there moving your comments so far and then we can let other people know that they can join and share their insights?

What do you think?

I have never used that feature so hopefully it's easy to use - please let me know if you run into any issues

from ml-engineering.

but you're starting a discussion, yes? We can discuss your questions there, then and invite others to contribute. That'd be much more productive than just you and I talking.

from ml-engineering.