[Image Captioning] out of memory immediately after training starts? about pytorch-tutorial HOT 16 CLOSED

I remember that the required gpu memory for batch_size=128 is less than 5GB (it may be much smaller).

What's your Python and PyTorch version? I guess you are using Python 2.7. Am i right?
This memory issue occurred when requires_grad=False does not work.

There are two options for solving this problem.

1. Upgrade PyTorch version
2. Use Python 3.5 instead of 2.7

from pytorch-tutorial.

This might be related to #26

This is a known issue which will be resolved in the next release.

Till then as a workaround, just change L56 to

               images = Variable(images, volatile=True)

and L66 to

                features = encoder(images)
                features = Variable(features.data)

from pytorch-tutorial.

@yunjey I am on python 2.7 and pytorch 0.12. I will try your changes.
@karandwivedi42 I will also test your fix.
I will let you both know.

from pytorch-tutorial.

@jtoy I recommend you to install PyTorch using source. This will give you the latest version of PyTorch.

from pytorch-tutorial.

I tried with pytorch python 2.7 source and using pytorch for python 3.5, both died with the same issue.

from pytorch-tutorial.

@karandwivedi42 your changes work! @yunjey will the code need to be updated? It seems like source doesnt seem to fix the issue. I can do more testing if needed.

from pytorch-tutorial.

@jtoy Ok. Thanks.

from pytorch-tutorial.

@karandwivedi42 That does not work.

images = Variable(images, volatile=True)

The code above makes requires_grad=False in resnet.fc. See here for the details of volatile.

from pytorch-tutorial.

@yunjey You are right. I don't know how important it is though because this linear layer is followed by another linear layer in the decoder with no non-linearity in between.

from pytorch-tutorial.

so what is the right code to use? I was able to train a model with @karandwivedi42 's change and the model completed training for me in 155 minutes. does that time seem right? I trained the original show and tell model and I remember it taking at least a day.

from pytorch-tutorial.

from pytorch-tutorial.

@karandwivedi42 I dont fully understand, Im just starting to play with pytorch, any way to see it as a diff ?

from pytorch-tutorial.

@jtoy This fork is a very hacky way to do exactly what the original code does.
https://github.com/karandwivedi42/pytorch-tutorial/tree/master/tutorials/09%20-%20Image%20Captioning

@yunjey Can you please check this one? (Thanks for the amazing tutorials btw :) )

from pytorch-tutorial.

@jtoy @karandwivedi42 I will fix the code by this weekend.

from pytorch-tutorial.

from pytorch-tutorial.

@jtoy @karandwivedi42 I modified the code. Try it. Thanks :)

from pytorch-tutorial.