NaN values at Multibox encoding about ssd.pytorch HOT 14 CLOSED

I was loading incorrectly the bounding boxes

from ssd.pytorch.

@andfoy - What was your fix?

Were you getting this same error:

THCudaCheck FAIL file=/b/wheel/pytorch-src/torch/lib/THC/generic/THCTensorMath.cu line=226 error=59 : device-side assert triggered
Traceback (most recent call last):
  File "train_cars.py", line 232, in <module>
    train()
  File "train_cars.py", line 184, in train
    loss_l, loss_c = criterion(out, targets)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mshah/code/ssd.pytorch/layers/modules/multibox_loss.py", line 70, in forward
    match(self.threshold,truths,defaults,self.variance,labels,loc_t,conf_t,idx)
  File "/home/mshah/code/ssd.pytorch/layers/box_utils.py", line 107, in match
    loc = encode(matches, priors, variances)
  File "/home/mshah/code/ssd.pytorch/layers/box_utils.py", line 133, in encode
    return torch.cat([g_cxcy, g_wh], 1)  # [num_priors,4]
RuntimeError: cuda runtime error (59) : device-side assert triggered at /b/wheel/pytorch-src/torch/lib/THC/generic/THCTensorMath.cu:226

from ssd.pytorch.

@andfoy @meetshah1995 I am also getting the same error as meetshah1995. Have you guys solve the problem?

from ssd.pytorch.

I forgot to load the bounding box annotations scaled by the image dimensions

from ssd.pytorch.

So the code cannot be used straight away? It must be corrected?

from ssd.pytorch.

No, you need to transform your annotations (Bounding boxes) into [x0/w, y0/h, x1/w, y1/h], where (x0, y0) and (x1, y1) are the coordinates of the upper left and lower right corners of the bounding box. h and w are the image height and width, respectively

from ssd.pytorch.

Isn't AnnotationTransform doing that?

from ssd.pytorch.

I think those transforms are loading the Pascal VOC XML data representation and then it transforms it to the representation described above. Anyway, if you are dealing with your own dataset, I would recommend to write your own loading routines based on the Pascal ones and then test them on the model. It is important that the bounding box annotations are on the correct format :-)

from ssd.pytorch.

Indeed, the easiest way by far is to convert your own dataset format into the Pascal VOC one (in a first instance).

from ssd.pytorch.

@andfoy @Cadene I am just trying to train Pascal and not my own dataset. So using AnnotationTransform should do the job? But I am still getting NaN. Can I know what I should change?

from ssd.pytorch.

@yxchng I am sorry, I don't know why you're getting NaN. Training on Pascal VOC 0712 was straightforward to me (I did not change anything). I am running the last version of pytorch available on conda. Did you try to reinstall pytorch?

from ssd.pytorch.

@Cadene It works with python3 but not with python2. I wonder why. Also, can I know how does your losses look like during training?

from ssd.pytorch.

@yxchng Hi, I just faced the same problems just like you in python2. Have you got some conclusions about the NaN loss error?

from ssd.pytorch.

@yxchng oh i see, the reason is '/' all the bounding box params will be 0; modify just like:
cur_pt = cur_pt / width if i % 2 == 0 else cur_pt / height
=>
cur_pt = 1.0 * cur_pt / width if i % 2 == 0 else 1.0 * cur_pt / height

from ssd.pytorch.