Error occur when training my own datasets without pretrained ,loss because nan about keras-yolo3 HOT 10 CLOSED

Try training with pretrained weights several epochs first, or starting training with a much small learning rate in the first several epochs. I think it is a problem of exp overflow.

from keras-yolo3.

@Ssrzbao Hi, do you solve your problem about nan now?

from keras-yolo3.

Same issue here, I will try a lower LR

from keras-yolo3.

I also get a bunch of warning right after starting the training:
/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py:3473: UserWarning: Skipping loading of weights for layer conv2d_59 due to mismatch in shape ((1, 1, 1024, 21) vs (255, 1024, 1, 1)).
weight_values[i].shape))
/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py:3473: UserWarning: Skipping loading of weights for layer conv2d_59 due to mismatch in shape ((21,) vs (255,)).
weight_values[i].shape))
/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py:3473: UserWarning: Skipping loading of weights for layer conv2d_67 due to mismatch in shape ((1, 1, 512, 21) vs (255, 512, 1, 1)).
weight_values[i].shape))
/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py:3473: UserWarning: Skipping loading of weights for layer conv2d_67 due to mismatch in shape ((21,) vs (255,)).
weight_values[i].shape))
/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py:3473: UserWarning: Skipping loading of weights for layer conv2d_75 due to mismatch in shape ((1, 1, 256, 21) vs (255, 256, 1, 1)).
weight_values[i].shape))
/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py:3473: UserWarning: Skipping loading of weights for layer conv2d_75 due to mismatch in shape ((21,) vs (255,)).
weight_values[i].shape))

from keras-yolo3.

I tried lr of 0.0001 but still went into nan quite fast. I think it has something to do with the layers not being properly initialized (see warnings above)

from keras-yolo3.

@ifeherva Did you solve the problem? I also met it with my own dataset.

from keras-yolo3.

@ifehervaI got the same problem when training my own data, did u solve the problem about the mismatch in shape??

from keras-yolo3.

@ifeherva @Ssrzbao Did you solve the problem yet?
I got same issue with coco dataset. Already used a low learning rate, but the loss value went to nan very soon and never came back...

from keras-yolo3.

@qqwweee Why are you closing this? I'm getting the same problem and so far I don't see any solution

from keras-yolo3.

@elcronos fixed in b7cbd24. The reason is displayed.

from keras-yolo3.