Out of memory while executing loss.backward() about ban-vqa HOT 6 CLOSED

It seems that torch.cuda.empty_cache() takes most of the time. I change the 81th line of train.py into total_loss += loss.data[0] * v.data.size(0) and remove torch.cuda.empty_cache(). And now it takes ~7450s/epoch (still quite slow...) and ~12G GPU memory. Maybe it's because pytorch's inefficient GPU memory occupation.

from ban-vqa.

Did you update NVIDIA drivers to the latest version? Some version seems to inefficiently use GPU memory. Usually it should work with 4 Titan Xs and the batch size of 256.

from ban-vqa.

Thanks for your timely help!
I will update it to 410.78. Btw, what's the version of Nvidia drivers i n your machine? My machines' versions are 384.90 and 390.67, but both didn't work for me.

from ban-vqa.

@jaesuny, could you provide the information about your environment in the recent reproduction for this issue? Or, any thought?

from ban-vqa.

Adding del loss, pred, att and torch.cuda.empty_cache() after every iteration seems to work for me. It only occupies ~7G GPU memory while training. However, the traning speed slows down to 10000s/epoch. So i'm still looking for a better solution. Thx!

from ban-vqa.

@cengzy14 thanks for the tip!

from ban-vqa.