Comments (10)
Try training with pretrained weights several epochs first, or starting training with a much small learning rate in the first several epochs. I think it is a problem of exp overflow.
from keras-yolo3.
@Ssrzbao Hi, do you solve your problem about nan now?
from keras-yolo3.
Same issue here, I will try a lower LR
from keras-yolo3.
I also get a bunch of warning right after starting the training:
/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py:3473: UserWarning: Skipping loading of weights for layer conv2d_59 due to mismatch in shape ((1, 1, 1024, 21) vs (255, 1024, 1, 1)).
weight_values[i].shape))
/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py:3473: UserWarning: Skipping loading of weights for layer conv2d_59 due to mismatch in shape ((21,) vs (255,)).
weight_values[i].shape))
/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py:3473: UserWarning: Skipping loading of weights for layer conv2d_67 due to mismatch in shape ((1, 1, 512, 21) vs (255, 512, 1, 1)).
weight_values[i].shape))
/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py:3473: UserWarning: Skipping loading of weights for layer conv2d_67 due to mismatch in shape ((21,) vs (255,)).
weight_values[i].shape))
/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py:3473: UserWarning: Skipping loading of weights for layer conv2d_75 due to mismatch in shape ((1, 1, 256, 21) vs (255, 256, 1, 1)).
weight_values[i].shape))
/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py:3473: UserWarning: Skipping loading of weights for layer conv2d_75 due to mismatch in shape ((21,) vs (255,)).
weight_values[i].shape))
from keras-yolo3.
I tried lr of 0.0001 but still went into nan quite fast. I think it has something to do with the layers not being properly initialized (see warnings above)
from keras-yolo3.
@ifeherva Did you solve the problem? I also met it with my own dataset.
from keras-yolo3.
@ifehervaI got the same problem when training my own data, did u solve the problem about the mismatch in shape??
from keras-yolo3.
@ifeherva @Ssrzbao Did you solve the problem yet?
I got same issue with coco dataset. Already used a low learning rate, but the loss value went to nan very soon and never came back...
from keras-yolo3.
@qqwweee Why are you closing this? I'm getting the same problem and so far I don't see any solution
from keras-yolo3.
@elcronos fixed in b7cbd24. The reason is displayed.
from keras-yolo3.
Related Issues (20)
- AttributeError: 'str' object has no attribute 'decode' HOT 3
- ImportError: cannot import name 'get_config' from 'tensorflow.python.eager.context'??
- How can I convert a keras model to yolov3 .weights file?
- How do you test model on data? HOT 1
- > 你好,我训练自己的数据集,win10+navida 3090+python3.6;执行train.py后,loss一直是nan,如何解决?请教指导,谢谢。
- width and height are resized to 416x416
- evaluate Model
- "Gradients do not exist" Warning
- "Epoch 00091:early stopping"It stops automatically when we get to step 90 HOT 1
- Does the model itself resize the image?
- low fps and no bounding box on Tensorflow 2 HOT 5
- AttributeError: 'str' object has no attribute 'decode' HOT 2
- how to use cudnn 7.1.4 ? HOT 2
- why the 2nd element of the yield in data_generator is of size equal to batch_size? HOT 1
- Training HOT 3
- Error when Data augmentation removed! Need help
- Questions about train model
- Adding metrics to the model HOT 1
- Error in loading annotaion labels while training HOT 1
- Encoding H264 not recognise after building open opencv manually
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from keras-yolo3.