reacher-l / 2020-ai-road-segmentation Goto Github PK

=> loading pretrained model True
=> loading pretrained model True
5069 images are loaded!
1157 images are loaded!
config: /backbone=hrnet/warmup=100_lr=0.01_multiplier=100_eta_min=0.0005_num_epochs=150_batchsize=4
Epoch [1]/[150] lr=0.010000
0%| | 0/1267 [00:00<?, ?it/s]THCudaCheck FAIL file=/pytorch/aten/src/THC/THCCachingHostAllocator.cpp line=296 error=77 : an illegal memory access was encountered
0%| | 0/1267 [00:02<?, ?it/s]
Traceback (most recent call last):
File "train.py", line 267, in
train_model()
File "train.py", line 152, in train_model
loss = calc_loss(outputs, labels, edge, metrics)
File "/home/sankuai/road_extra/loss.py", line 446, in calc_loss
loss_ls = lovasz_softmax(F.softmax(pred[1],dim=1),target)
File "/home/sankuai/road_extra/loss.py", line 185, in lovasz_softmax
loss = lovasz_softmax_flat(*flatten_probas(probas, labels, ignore), classes=classes)
File "/home/sankuai/road_extra/loss.py", line 204, in lovasz_softmax_flat
if (classes is 'present' and fg.sum() == 0):
RuntimeError: CUDA error: an illegal memory access was encountered