Multi GPU process stuck in the dataloader loop, single GPU works fine. about accelerate HOT 7 CLOSED

Why are we combining accelerate then? I'm not sure those two can exist at the same time. This feels like a torchkeras issue more than accelerate.

Thanks, I find the solution...This is a bug of Pytorch. In 'torch.utils.data.sampler', we must use "torch.randperm" method, which does not work in a multiprocess environment when passed a value greater than 2**15. So I reduce the size of my dataset, and it works!!!

from accelerate.

When I use debug mode of Pycharm, I find that the procedure will stuck in the dataloader loop.

from accelerate.

You may need to set the env variable inside the training function. Can you share your full notebook code?

from accelerate.

You may need to set the env variable inside the training function. Can you share your full notebook code?

Thank you for your prompt reply, it doesn't work as well. And the code above is all I have. I'm trying to standardize my code with torchkeras.

from accelerate.

Not sure if we support TorchKeras models. Does isinstance(model, torch.nn.Module) return True?

from accelerate.

isinstance(model, torch.nn.Module)

yes, it returns Ture. And I think we can think of torchkeras as a normal training program.

from accelerate.

Why are we combining accelerate then? I'm not sure those two can exist at the same time. This feels like a torchkeras issue more than accelerate.

from accelerate.