Pytorch lighting Fabric + lit data + DDP hangs when finishing epoch about litdata HOT 3 OPEN

Hey @miguelalba96. Thanks for reporting this issue.

Would you mind printing the length of each dataset, dataloader on each rank. Usually it hangs when a rank have more data than others. It shouldn't happen but I want to exclude this eventuallity.

Do you think you could share a tiny reproducible example with dummy data for me to debug ?

Best,
T.C

from litdata.

when printing the ranks per node and on each: len(dataset), len(dataloader)` I get homogeneous number of samples on each:

Not sure how to reproduce this problem, I will check. I also noticed that when I load the state to resume training using the function I wrote above get_state, the dataloader doesn't seem to resume properly and iterates all over again through the data until it hangs 🤔:

from litdata.

Hey @miguelalba96, any chance you could create a reproducible Studio on https://lightning.ai/ that I can duplicate to investigate what's happening. Otherwise, it is hard for me to help you.

from litdata.