My trainig process is frozen about fots.pytorch HOT 5 OPEN

Did u use DDP training? If so, may be spawned process might occur some errors causing it to quit. So the main process was still waiting.

from fots.pytorch.

Yeah, I fixed trainer like this to fit the version of Lightning(currently using with single gpu):

trainer = Trainer(
        logger=wandb_logger,
        callbacks=[checkpoint_callback],
        max_epochs=config.trainer.epochs,
        default_root_dir=root_dir,
        devices=gpus,
        accelerator='cuda',
        benchmark=True,
        sync_batchnorm=True,
        precision=config.precision,
        log_every_n_steps=config.trainer.log_every_n_steps,
        overfit_batches=config.trainer.overfit_batches,
        fast_dev_run=config.trainer.fast_dev_run,
        inference_mode=True,
        check_val_every_n_epoch=config.trainer.check_val_every_n_epoch,
        strategy=DDPStrategy(find_unused_parameters=True)
        )

Then is "DDPStrategy" related to this problem? I'm new at using Lightning, so how can I fix it?

from fots.pytorch.

You can use CPU training or single GPU training mode to check if any bad thing occurs.

from fots.pytorch.

I already did with single GPU and got no problem.
Then Isn't there any solution with multiple GPUs?

from fots.pytorch.

try to set the worker number of Dataset to 0 and see.

from fots.pytorch.