dataloader runtime errors about tsn-pytorch HOT 20 CLOSED

IF I set workers = 0, this error will disappear.

from tsn-pytorch.

Are you using the latest version of PyTorch?

from tsn-pytorch.

from tsn-pytorch.

You only have one GPU, use -j 1. A too high -j number or a too small GPU men size may cause this error.

from tsn-pytorch.

And you are using a Batchsize of 128 with only 1 GPU. I don’t think this is feasible. It typically needs 4 GPUs.

from tsn-pytorch.

@yjxiong Actually I have 4 GPUs, if I set j=1, it will stuck there. Only if j=0 works.

from tsn-pytorch.

I have explained why your case failed. Try use 4 GPUs instead.

from tsn-pytorch.

I have removed the --gpus option, it will automatically use 4 GPUs. It does not help.

from tsn-pytorch.

What is the memory size of your GPUs? If setting -j 4 or lower won’t work, I don’t know what’s the error then. We run this setting on 4 Titan X without problem.

from tsn-pytorch.

from tsn-pytorch.

Then I don’t have any instant idea for this. Seeing from the log, it says multiprocessing lib in the dataloaders cannot open/write to some kind of shared momery for communication between processes. It may be a problem in your permission setting or os, which I cannot identify on my side.

from tsn-pytorch.

from tsn-pytorch.

I don’t think this is a good choice tbh. It wastes too much time for data loading.

I’d rather suggest you either figure out the problem or go with Caffe TSN instead. I don’t want you to blame the code for running super slow at the end of the day.

from tsn-pytorch.

from tsn-pytorch.

I find it is caused by lack of shared memory.
I have increased the shared memory size and it can be trained using 8 workers!

from tsn-pytorch.

Good to know. This should lead to reasonable training speed.

from tsn-pytorch.

I met the same probles as you. Could you explain carefully how to solve this problem? I have no idea about the "lack of shared memory."
thank you

from tsn-pytorch.

@JiqiangZhou Please type in this command "df –k /dev/shm".
You will see the shared memory size in your PC or server.
Then if it is small, you will need to enlarge it.

from tsn-pytorch.

@ntuyt available is only 65536, I think it's too small! Iwill try to increase it. Thank you.

from tsn-pytorch.

I tried many times and the space of /dev/shm is at least 35GB.

from tsn-pytorch.