Strange errors when training with 2x GPUs about cct HOT 3 CLOSED

@PkuRainBow If I remember correctly; the results are quite similar without the need for sync bn.

from cct.

Hi @PkuRainBow, thank you for your interest in our work.

This error is at the end of the first epoch, where in the last batch, you have only one image and BN gives an error, to solve this, please simply set drop_last = True in the dataloaders (or change the batch size / do an if statement to skip the last step if BZ = 1).

from cct.

@yassouali Thanks for your quick reply. I am wondering whether you have compared the final performance of the method trained with 1x 32G V10 GPU and the method trained with 2x 16G V100 GPUs as we want to accelerate the training time.
Especially, we are not sure whether SyncBN is necessary when using 2x GPUs.

from cct.