GPU utilization falls back to 0% when training with multiple GPUs about electra_pytorch HOT 5 CLOSED

Preprocessing of data is precomupted and cached, the tokenizer is just to provide pad_id and other attributes afterwards, so tokenizer might not be the problem.

I haven't tried data parallelism yet, so I have no good advises for you. But I will try to manually measure the time of each part or use Pytorch profiler if I got problem like this.

from electra_pytorch.

It mightbe that the reason is that my corpus is very small.
I think it is so small that it gets computed in 4 batches.
After each epoch it needs some time because of the checks it does to maybe save the checkpoint.
I will report back when I know more.

from electra_pytorch.

Is c.steps the real number of epochs? Or is it steps (batches)?

from electra_pytorch.

actual number of batches to compute

from electra_pytorch.

Ok thanks.

When I am using more training data the utilization is much more stable.

Closing this.

from electra_pytorch.