"RuntimeError: reduce failed to synchronize: device-side assert triggered" while trainging about yolov3-network-slimming HOT 8 CLOSED

I met the same problem. The training log show that the width is very very large.

from yolov3-network-slimming.

@MrWangg1992 Hi, how do you solve the this problem? Can you share it with me?

from yolov3-network-slimming.

@violet17 I changed the BCELoss command to BCEWithLogitsLoss

from yolov3-network-slimming.

@MrWangg1992 Thanks. I changed it too, but it can't help. loss became nan.

from yolov3-network-slimming.

@violet17 Did you changed the batch and subdivison size in the .cfg file as well ?

from yolov3-network-slimming.

@MrWangg1992 I changed them to 1 because of CUDA always out of memory.

from yolov3-network-slimming.

@violet17 你看看你的cfg配置文件选的是test还是train，batch和subdivisions等于1的时候是没有效果的. Out of memory 就只能调batch size 这些的调小点

from yolov3-network-slimming.

@MrWangg1992 我把batch和subdivisions都设成1了， 不能这样么？为啥？

from yolov3-network-slimming.