RuntimeError: cuda runtime error (2) : out of memory at... about munit HOT 9 OPEN

Hi
You can change the display_size in the config file to 4 or 2, and try again.

from munit.

@visonpon I couldn't fix this. Could you please provide details? I have the same problem and I'm new to torch.

UPDATE: Ok I got it, just read the cmd error output of pytorch, it tells what to do.
Thx.

from munit.

@WuChanghao233 Still... i change display_size in the config file (changed to 2)

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1524580938250/work/aten/src/THC/THCTensorRandom.cu line=25 error=2 : out of memory
Traceback (most recent call last):
File "train.py", line 5, in
from utils import get_all_data_loaders, prepare_sub_folder,
File "/root/model/utils.py", line 5, in
from torch.utils.serialization import load_lua
File "/root/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/init.py", line 2, in
from .read_lua_file import load_lua, T7Reader
File "/root/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 184, in
register_torch_class('Storage', make_storage_reader)
File "/root/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 181, in register_torch_class
reader_registry[cls_name] = reader_factory(cls_name)
File "/root/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 160, in make_storage_reader
element_size = python_class().element_size()
File "/root/anaconda2/lib/python2.7/site-packages/torch/cuda/init.py", line 492, in _lazy_new
_lazy_init()
File "/root/anaconda2/lib/python2.7/site-packages/torch/cuda/init.py", line 161, in _lazy_init
torch._C._cuda_init()
RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1524580938250/work/aten/src/THC/THCTensorRandom.cu:25

from munit.

Same problem occurs to me, my GPU has 11GB memory, but it can't train nether either.

from munit.

Same problem occurs to me, my GPU is GTX 1050Ti (4GB memory).
I try display_size in the config file to 4 or 2, not solve it.

from munit.

i have also encountered this problem , it seems like the part of write_image script lead to this bug since when i commented it , everything is ok. but i don't know how to fix it when i want to see the result during training @mingyuliutw

from munit.

solved it by adding with torch_no_grad:

from munit.

solved it by adding with torch_no_grad:

Can you tell me where to add it?

from munit.

Adjusting the .yaml to shrink/simplify the network works. I imagine the only other options are to optimize their code for memory usage or get a better GPU.

from munit.