RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/ about torchseg HOT 8 CLOSED

26 21:25:43 PyTorch Version 1.0.1.post2, Furnace Version 0.1.1
26 21:25:43 PyTorch Version 1.0.1.post2, Furnace Version 0.1.1
26 21:25:43 PyTorch Version 1.0.1.post2, Furnace Version 0.1.1
26 21:25:43 PyTorch Version 1.0.1.post2, Furnace Version 0.1.1
[00:00<?,?it/s]Traceback (most recent call last):
  File "train.py", line 123, in <module>
    loss = model(imgs, gts)
  File "/home/USER/BiSeNet-official-env/lib/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/USER/BiSeNet-official-env/lib/python3.5/site-packages/apex-0.1-py3.5-linux-x86_64.egg/apex/parallel/distributed.py", line 459, in forward
    result = self.module(*inputs, **kwargs)
  File "/home/USER/BiSeNet-official-env/lib/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/USER/Projects/TorchSeg-BiSeNet/model/bisenet/cityscapes.bisenet.R18/network.py", line 105, in forward
    aux_loss0 = self.ohem_criterion(self.heads[0](pred_out[0]), label)
  File "/home/USER/BiSeNet-official-env/lib/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/USER/Projects/TorchSeg-BiSeNet/furnace/seg_opr/loss_opr.py", line 84, in forward
    index = mask_prob.argsort()
  File "/home/USER/BiSeNet-official-env/lib/python3.5/site-packages/torch/tensor.py", line 248, in argsort
    return torch.argsort(self, dim, descending)
  File "/home/USER/BiSeNet-official-env/lib/python3.5/site-packages/torch/functional.py", line 648, in argsort
    return torch.sort(input, -1, descending)[1]
RuntimeError: CUDA error: device-side assert triggered

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 157, in <module>
    config.log_dir_link)
  File "/home/USER/Projects/TorchSeg-BiSeNet/furnace/engine/engine.py", line 154, in __exit__
    torch.cuda.empty_cache()
  File "/home/USER/BiSeNet-official-env/lib/python3.5/site-packages/torch/cuda/__init__.py", line 374, in empty_cache
    torch._C._cuda_emptyCache()
RuntimeError: CUDA error: device-side assert triggered

Exception ignored in: <bound method Event.__del__ of <torch.cuda.Event 0x34cf360>>
Traceback (most recent call last):
  File "/home/USER/BiSeNet-official-env/lib/python3.5/site-packages/torch/cuda/streams.py", line 167, in __del__
  File "/home/USER/BiSeNet-official-env/lib/python3.5/site-packages/torch/cuda/__init__.py", line 208, in check_error
torch.cuda.CudaError: device-side assert triggered (59)
terminate called without an active exception

diff --git a/furnace/datasets/BaseDataset.py b/furnace/datasets/BaseDataset.py
index 7d8f6ef..bfcaa6b 100755
--- a/furnace/datasets/BaseDataset.py
+++ b/furnace/datasets/BaseDataset.py
@@ -10,6 +10,7 @@ import time
 import cv2
 import torch
 import numpy as np
+from skimage.segmentation import relabel_sequential
 
 import torch.utils.data as data
 
@@ -62,6 +63,15 @@ class BaseDataset(data.Dataset):
         if self.preprocess is not None and extra_dict is not None:
             output_dict.update(**extra_dict)
 
+        gt = gt.cpu().numpy()
+        trans_labels = [7, 8, 11, 12, 13, 17, 19, 20, 21, 22, 23, 24, 25, 26, 27,
+                        28, 31, 32, 33, 0]
+        trans_labels = np.array(trans_labels)
+        _, fw, inv = relabel_sequential(trans_labels)
+        gt[gt==255] = 0
+        gt = fw[gt]
+        gt = torch.from_numpy(np.ascontiguousarray(gt)).long()
+        output_dict['label'] = gt
         return output_dict
 
     def _fetch_data(self, img_path, gt_path, dtype=None):