@RickvanHek Sorry, its possible one of the latest commits broke something, as I've been updating quite often. I've been running internal tests with a more advanced version with no issues. I just committed these changes now as b07ee41. Can you try to download this from scratch and rerun?

If this works I'd advise you to check back in a day or two, I'm compiling updates that should increase performance significantly. These updates are not reflected in the latest commit, they are being tracked in #2 (comment) and should be implemented in one single commit in the next few days.

FYI in your particular case you may have issues with your local dataset with SGD burn-in, which is supposed to last the first 1000 batches (you only have 99). So your first epoch may be doing nothing really, as the lr is programmed to ramp exponentially from zero to the initial lr = 0.001 over those 1000 batches.

from yolov3.

@RickvanHek Sorry, its possible one of the latest commits broke something, as I've been updating quite often. I've been running internal tests with a more advanced version with no issues. I just committed these changes now as b07ee41. Can you try to download this from scratch and rerun?

If this works I'd advise you to check back in a day or two, I'm compiling updates that should increase performance significantly. These updates are not reflected in the latest commit, they are being tracked in #2 (comment) and should be implemented in one single commit in the next few days.

FYI in your particular case you may have issues with your local dataset with SGD burn-in, which is supposed to last the first 1000 batches (you only have 99). So your first epoch may be doing nothing really, as the lr is programmed to ramp exponentially from zero to the initial lr = 0.001 over those 1000 batches.
yolov3/train.py

Lines 139 to 143 in b07ee41

SGD burn-in

if (epoch == 0) & (i <= 1000):
lr = lr0 * (i / 1000) ** 4
for g in optimizer.param_groups:
g['lr'] = lr

Thank you for the fast reply, when I disable the SGD (on the last commit, like you said), I don't get the nan anymore, if i enable it i still get these nan's. These are my results after 4 epochs,

      Epoch      Batch          x          y          w          h       conf        cls      total          P          R   nTargets         TP         FP         FN       time
       0/99      99/99       1.31       1.28       3.92        8.5       9.03       7.27       31.3          0          0          3          0          0          0     0.0571
       1/99      99/99       1.21       1.11       1.04       1.34        3.3       7.27       15.3          0          0         10          0          0          0      0.115
       2/99      99/99       1.16       1.09       1.07       1.28       2.84       7.19       14.6          0          0          7          0          0          0      0.116
       3/99      99/99       1.04      0.963      0.891       1.09       2.61       7.21       13.8          0          0         10          0          0          0      0.115

My dataset has 70000 training and validation images, I was just using a 100 for testing purpose.
I am now running it on 7000 training images to see if it'll work.

from yolov3.

I did 6 epochs on 7000 images (batch size = 1), and i still got 0 precision, here are the results:

      Epoch      Batch          x          y          w          h       conf        cls      total          P          R   nTargets         TP         FP         FN       time
       0/99  6999/6999       0.33      0.315      0.789      0.896       6.69       6.02         15          0          0         10          0          0          0     0.0937
       1/99  6999/6999      0.144      0.133      0.474      0.463      0.836       3.41       5.46          0          0          7          0          0          0      0.116
       2/99  6999/6999     0.0981     0.0887       0.44      0.376      0.637       1.93       3.57          0          0          6          0          0          0      0.116
       3/99  6999/6999     0.0824     0.0764       0.39      0.321      0.576       1.13       2.58          0          0          6          0          0          0      0.121
       4/99  6999/6999     0.0725      0.065      0.347      0.261      0.522      0.662       1.93          0          0         10          0          0          0      0.117
       5/99  6999/6999     0.0666     0.0592      0.285      0.213      0.495      0.428       1.55          0          0          4          0          0          0      0.131

Any idea why i'm getting 0 precision? Left the code untouched only edited the writing to result.txt (to not do an mAp calculation, because this gives an error for me)

from yolov3.

Ah that's perfect, everything is fine. The new -batch_report flag in train.py controls whether P, R, TP, FP, and FN print to screen during training. Printing to screen makes training about 20% slower so I set it to False by default, but you can turn this back on to see those print to screen.

The mAP calculation in test.py now adds P and R to results.txt in the last 3 columns. Is test.py still crashing?

from yolov3.

Ah that's perfect, everything is fine. The new -batch_report flag in train.py controls whether P, R, TP, FP, and FN print to screen during training. Printing to screen makes training about 20% slower so I set it to False by default, but you can turn this back on to see those print to screen.

The mAP calculation in test.py now adds P and R to results.txt in the last 3 columns. Is test.py still crashing?

Oh that's why, didn't know that.
And yes it still crashing, this is my error:

Traceback (most recent call last):
  File "/home/rick/Documents/yolov3/train.py", line 218, in <module>
    main(opt)
  File "/home/rick/Documents/yolov3/train.py", line 209, in main
    mAP, R, P = test.main(test.opt)
  File "/home/rick/Documents/yolov3/test.py", line 42, in main
    model.load_state_dict(checkpoint['model'])
  File "/media/rick/HDD/Env/latestyolov3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 719, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for Darknet:
	size mismatch for module_list.81.conv_81.weight: copying a param of torch.Size([255, 1024, 1, 1]) from checkpoint, where the shape is torch.Size([303, 1024, 1, 1]) in current model.
	size mismatch for module_list.81.conv_81.bias: copying a param of torch.Size([255]) from checkpoint, where the shape is torch.Size([303]) in current model.
	size mismatch for module_list.93.conv_93.weight: copying a param of torch.Size([255, 512, 1, 1]) from checkpoint, where the shape is torch.Size([303, 512, 1, 1]) in current model.
	size mismatch for module_list.93.conv_93.bias: copying a param of torch.Size([255]) from checkpoint, where the shape is torch.Size([303]) in current model.
	size mismatch for module_list.105.conv_105.weight: copying a param of torch.Size([255, 256, 1, 1]) from checkpoint, where the shape is torch.Size([303, 256, 1, 1]) in current model.
	size mismatch for module_list.105.conv_105.bias: copying a param of torch.Size([255]) from checkpoint, where the shape is torch.Size([303]) in current model.

from yolov3.

Ok. Layers 81, 93 and 105 are the yolo detection layers. 80 classes makes them size 255 but your model seems to have more classes as you can see.

If you are using a custom dataset you need to adjust test.py to work with it, just like you adjust train.py (i.e. set -class_path, -cfg, -data_config_path correctly), otherwise yes, it will crash.

from yolov3.

@RickvanHek I just had an idea that might smooth things a bit after seeing #40 recently. If I pass the train.py namespace arguments (-class_path, -cfg, -data_config_path) to test.py then you wouldn't need to define them twice (during training).

If you ran test.py by itself you'd still need to properly set them again though. Do you think this change might have solved your training issues?

from yolov3.

@RickvanHek I just had an idea that might smooth things a bit after seeing #40 recently. If I pass the train.py namespace arguments (-class_path, -cfg, -data_config_path) to test.py then you wouldn't need to define them twice (during training).

If you ran test.py by itself you'd still need to properly set them again though. Do you think this change might have solved your training issues?

Hello, this fixed the problem! Thank you!

from yolov3.