multi gpu training hangs in python threading about few-shot-vid2vid HOT 4 CLOSED

Maybe your "--batchSize 224" is too large, and the original batchSize is 60 with 8 gpus.
You can try a small batchSize.

from few-shot-vid2vid.

@iloveOREO Thank you for your kind help!
However I tried to lower the batch size, the problem still exist
It's So strange, I just can't figure out what could be wrong
And if I'm just 'lucky' enough, the code can run without any problem sometimes :(

from few-shot-vid2vid.

@MengXinChengXuYuan You're welcome!
I think the error maybe raise by multi-process, which may under the influence of param "--nThreads".
I am using "pytorch/pytorch:1.2-cuda10.0-cudnn7-devel" docker image and it works fine with me.
I had run example with given pose dataset on Tesla P100 by "python train.py --name pose --dataset_mode fewshot_pose
--adaptive_spade --warp_ref --spade_combine --remove_face_labels --add_face_D
--niter_single 100 --niter 200
--gpu_ids 0,1 --batchSize 4 --nThreads 2 --continue_train "
Hope it can be helpful.

from few-shot-vid2vid.

@iloveOREO I see! Thank you for your help :p
I will try to lower the threads I use

from few-shot-vid2vid.