Comments (22)
MichaelMonashev
commented on July 28, 2024
1
I start training with debug on full dataset. May be after 2 days I get some useful information.
from keras-maskrcnn.
vcarpani
commented on July 28, 2024
It looks like there is an empty mask in your validation set
from keras-maskrcnn.
MichaelMonashev
commented on July 28, 2024
It works 10 epoches and everything was ok.
Bad mask or bbox had to fail on first epoch.
from keras-maskrcnn.
MichaelMonashev
commented on July 28, 2024
my dataset was generated using rendering software. I calculated bboxes from masks:
mask = cv2.imread(maskPath, cv2.IMREAD_GRAYSCALE)
where = np.array(np.where(mask>127))
if where.size > 0:
y1, x1 = np.amin(where, axis=1)
y2, x2 = np.amax(where, axis=1)
from keras-maskrcnn.
hgaiser
commented on July 28, 2024
Strange that the error occurs after several epochs, but that error does usually indicate that a file isn't properly loaded. You could try printing type(mask) before the line where it crashes. I'm guessing it will say NoneType just before it crashes. Maybe if you also print the file where it crashes on, you get some more perspective of what is happening. Maybe it's always happening on the same mask?
from keras-maskrcnn.
MichaelMonashev
commented on July 28, 2024
This problem dispraiser then I add checks for very small masks:
if abs(x2-x1)<8 or abs(y2-y1)< 8 or (x2-x1)*(y2-y1)<400:
I think small mask, like one dot, can raise this error again.
from keras-maskrcnn.
hgaiser
commented on July 28, 2024
Hmm even small masks shouldn't cause a crash. Could you make a small dataset containing one image and one object in that image that crash? Then I can test on it further to fix it to ensure it won't crash.
from keras-maskrcnn.
MichaelMonashev
commented on July 28, 2024
Sorry, I cant do this. Its not public dataset.
from keras-maskrcnn.
hgaiser
commented on July 28, 2024
If you know how to make the issue happen, you can try to do it on a publicly available image?
from keras-maskrcnn.
MichaelMonashev
commented on July 28, 2024
I generated several masks, that may cause this error: http://i87.beon.ru/masks.tar .
from keras-maskrcnn.
MichaelMonashev
commented on July 28, 2024
again get this error. At the same epoch, after saving snapshot. I think, snapshot saving do something wrong. For example, corrupt information about bboxes.
1784 instances of class 111 with average precision: 0.8032
1745 instances of class 222 with average precision: 0.8715
1822 instances of class 333 with average precision: 0.8122
1814 instances of class 444 with average precision: 0.8563
1827 instances of class 555 with average precision: 0.8343
mAP: 0.8355
Epoch 9/50
10000/10000 [==============================] - 12373s 1s/step - loss: 0.5614 - regression_loss: 0.3762 - classification_loss: 0.0587 - masks_loss: 0.1265
Epoch 00009: saving model to /var/tmp/xxx/snapshots/resnet50_csv_09.h5
1784 instances of class 111 with average precision: 0.8047
1745 instances of class 222 with average precision: 0.8764
1822 instances of class 333 with average precision: 0.8262
1814 instances of class 444 with average precision: 0.8571
1827 instances of class 555 with average precision: 0.8250
mAP: 0.8379
Epoch 10/50
10000/10000 [==============================] - 12646s 1s/step - loss: 0.5286 - regression_loss: 0.3544 - classification_loss: 0.0538 - masks_loss: 0.1203
Epoch 00010: saving model to /var/tmp/xxx/snapshots/resnet50_csv_10.h5
1784 instances of class 111 with average precision: 0.8207
1745 instances of class 222 with average precision: 0.8744
1822 instances of class 333 with average precision: 0.8388
1814 instances of class 444 with average precision: 0.8477
1827 instances of class 555 with average precision: 0.8351
mAP: 0.8434
Epoch 11/50
10000/10000 [==============================] - 12794s 1s/step - loss: 0.4994 - regression_loss: 0.3351 - classification_loss: 0.0495 - masks_loss: 0.1148
Epoch 00011: saving model to /var/tmp/xxx/snapshots/resnet50_csv_11.h5
Traceback (most recent call last):
File "/home/xxxx/keras-maskrcnn/keras_maskrcnn/bin/train.py", line 313, in <module>
main()
File "/home/xxxx/keras-maskrcnn/keras_maskrcnn/bin/train.py", line 309, in main
max_queue_size=1,
File "/home/xxxx/.local/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/xxxx/.local/lib/python3.5/site-packages/keras/engine/training.py", line 1418, in fit_generator
initial_epoch=initial_epoch)
File "/home/xxxx/.local/lib/python3.5/site-packages/keras/engine/training_generator.py", line 251, in fit_generator
callbacks.on_epoch_end(epoch, epoch_logs)
File "/home/xxxx/.local/lib/python3.5/site-packages/keras/callbacks.py", line 79, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "/home/xxxx/.local/lib/python3.5/site-packages/keras_retinanet/callbacks/common.py", line 31, in on_epoch_end
self.callback.on_epoch_end(epoch, logs=logs)
File "/home/xxxx/keras-maskrcnn/keras_maskrcnn/bin/../../keras_maskrcnn/callbacks/eval.py", line 66, in on_epoch_end
save_path=self.save_path
File "/home/xxxx/keras-maskrcnn/keras_maskrcnn/bin/../../keras_maskrcnn/utils/eval.py", line 212, in evaluate
mask = cv2.resize(mask, (box[2] - box[0], box[3] - box[1]))
cv2.error: OpenCV(3.4.3) /io/opencv/modules/imgproc/src/resize.cpp:4045: error: (-215:Assertion failed) !dsize.empty() || (inv_scale_x > 0 && inv_scale_y > 0) in function 'resize'
Exception ignored in: <bound method BaseSession.__del__ of <tensorflow.python.client.session.Session object at 0x7f557ea91c18>>
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 696, in __del__
TypeError: 'NoneType' object is not callable
from keras-maskrcnn.
hgaiser
commented on July 28, 2024
again get this error. At the same epoch, after saving snapshot. I think, snapshot saving do something wrong. For example, corrupt information about bboxes.
Did you try my previous suggestion?
Strange that the error occurs after several epochs, but that error does usually indicate that a file isn't properly loaded. You could try printing
type(mask)before the line where it crashes. I'm guessing it will sayNoneTypejust before it crashes. Maybe if you also print the file where it crashes on, you get some more perspective of what is happening. Maybe it's always happening on the same mask?
from keras-maskrcnn.
MichaelMonashev
commented on July 28, 2024
No, I used my private dataset.
from keras-maskrcnn.
MichaelMonashev
commented on July 28, 2024
I added debug print(type(mask), box[2], box[0], box[3], box[1]) and started training on very small dataset with --steps=1 . It worked 50 epoches and finished without errors. Magic...
from keras-maskrcnn.
MichaelMonashev
commented on July 28, 2024
mAP: 0.3976
Epoch 2/50
10000/10000 [==============================] - 11666s 1s/step - loss: 0.7923 - regression_loss: 0.4931 - classification_loss: 0.1622 - masks_loss: 0.1370
Epoch 00002: saving model to /var/tmp/xxx/snapshots/resnet50_csv_02.h5
<class 'numpy.ndarray'> 994 913 881 840
<class 'numpy.ndarray'> 616 544 662 623
<class 'numpy.ndarray'> 372 286 882 839
<class 'numpy.ndarray'> 346 270 664 624
<class 'numpy.ndarray'> 618 544 620 583
<class 'numpy.ndarray'> 1162 1088 663 495
<class 'numpy.ndarray'> 529 455 658 489
<class 'numpy.ndarray'> 125 96 654 496
<class 'numpy.ndarray'> 138 105 651 495
<class 'numpy.ndarray'> 371 287 925 883
<class 'numpy.ndarray'> 498 427 908 751
<class 'numpy.ndarray'> 706 635 648 493
<class 'numpy.ndarray'> 164 133 649 497
...
<class 'numpy.ndarray'> 163 143 642 464
<class 'numpy.ndarray'> 156 151 632 468
<class 'numpy.ndarray'> 837 815 639 470
<class 'numpy.ndarray'> 619 601 635 460
<class 'numpy.ndarray'> 801 725 684 518
<class 'numpy.ndarray'> 132 119 631 466
<class 'numpy.ndarray'> 258 190 634 477
<class 'numpy.ndarray'> 1034 1008 936 767
<class 'numpy.ndarray'> 715 646 635 482
<class 'numpy.ndarray'> 533 457 598 558
<class 'numpy.ndarray'> 615 613 636 472
<class 'numpy.ndarray'> 569 572 637 472
Traceback (most recent call last):
File "/home/xxxxx/keras-maskrcnn/keras_maskrcnn/bin/train.py", line 313, in <module>
main()
File "/home/xxxxx/keras-maskrcnn/keras_maskrcnn/bin/train.py", line 309, in main
max_queue_size=1,
File "/home/xxxxx/.local/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/xxxxx/.local/lib/python3.5/site-packages/keras/engine/training.py", line 1418, in fit_generator
initial_epoch=initial_epoch)
File "/home/xxxxx/.local/lib/python3.5/site-packages/keras/engine/training_generator.py", line 251, in fit_generator
callbacks.on_epoch_end(epoch, epoch_logs)
File "/home/xxxxx/.local/lib/python3.5/site-packages/keras/callbacks.py", line 79, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "/home/xxxxx/.local/lib/python3.5/site-packages/keras_retinanet/callbacks/common.py", line 31, in on_epoch_end
self.callback.on_epoch_end(epoch, logs=logs)
File "/home/xxxxx/keras-maskrcnn/keras_maskrcnn/bin/../../keras_maskrcnn/callbacks/eval.py", line 66, in on_epoch_end
save_path=self.save_path
File "/home/xxxxx/keras-maskrcnn/keras_maskrcnn/bin/../../keras_maskrcnn/utils/eval.py", line 215, in evaluate
mask = cv2.resize(mask, (box[2] - box[0], box[3] - box[1]))
cv2.error: OpenCV(3.4.3) /io/opencv/modules/imgproc/src/resize.cpp:4045: error: (-215:Assertion failed) !dsize.empty() || (inv_scale_x > 0 && inv_scale_y > 0) in function 'resize'
from keras-maskrcnn.
MichaelMonashev
commented on July 28, 2024
box[2] - box[0] is less than zero.
from keras-maskrcnn.
MichaelMonashev
commented on July 28, 2024
I got it again:
<class 'numpy.ndarray'> 295 277 759 614
<class 'numpy.ndarray'> 768 706 761 728
<class 'numpy.ndarray'> 802 789 752 617
<class 'numpy.ndarray'> 574 551 750 620
<class 'numpy.ndarray'> 1095 1066 768 626
<class 'numpy.ndarray'> 993 923 763 614
<class 'numpy.ndarray'> 1061 999 761 729
<class 'numpy.ndarray'> 1244 1180 762 618
<class 'numpy.ndarray'> 696 668 765 639
<class 'numpy.ndarray'> 1049 1007 721 629
<class 'numpy.ndarray'> 1049 1007 758 669
<class 'numpy.ndarray'> 131 135 723 631
Traceback (most recent call last):
File "/home/xxxxx/keras-maskrcnn/keras_maskrcnn/bin/train.py", line 313, in <module>
main()
File "/home/xxxxx/keras-maskrcnn/keras_maskrcnn/bin/train.py", line 309, in main
max_queue_size=1,
File "/home/xxxxx/.local/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/xxxxx/.local/lib/python3.5/site-packages/keras/engine/training.py", line 1418, in fit_generator
initial_epoch=initial_epoch)
File "/home/xxxxx/.local/lib/python3.5/site-packages/keras/engine/training_generator.py", line 251, in fit_generator
callbacks.on_epoch_end(epoch, epoch_logs)
File "/home/xxxxx/.local/lib/python3.5/site-packages/keras/callbacks.py", line 79, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "/home/xxxxx/.local/lib/python3.5/site-packages/keras_retinanet/callbacks/common.py", line 31, in on_epoch_end
self.callback.on_epoch_end(epoch, logs=logs)
File "/home/xxxxx/keras-maskrcnn/keras_maskrcnn/bin/../../keras_maskrcnn/callbacks/eval.py", line 66, in on_epoch_end
save_path=self.save_path
File "/home/xxxxx/keras-maskrcnn/keras_maskrcnn/bin/../../keras_maskrcnn/utils/eval.py", line 215, in evaluate
mask = cv2.resize(mask, (box[2] - box[0], box[3] - box[1]))
cv2.error: OpenCV(3.4.3) /io/opencv/modules/imgproc/src/resize.cpp:4045: error: (-215:Assertion failed) !dsize.empty() || (inv_scale_x > 0 && inv_scale_y > 0) in function 'resize'
Exception ignored in: <bound method BaseSession.__del__ of <tensorflow.python.client.session.Session object at 0x7feaf3a51c50>>
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 696, in __del__
TypeError: 'NoneType' object is not callable
>
from keras-maskrcnn.
MichaelMonashev
commented on July 28, 2024
I look to the val.csv and can`t find bboxes with size like above.
from keras-maskrcnn.
hgaiser
commented on July 28, 2024
I look to the val.csv and can`t find bboxes with size like above.
It could be that these boxes are printed after reshaping.
from keras-maskrcnn.
RohitMidha23
commented on July 28, 2024
Hey @MichaelMonashev did you find a fix to this error : (inv_scale_x > 0 && inv_scale_y > 0) in function 'resize'?
from keras-maskrcnn.
MichaelMonashev
commented on July 28, 2024
@RohitMidha23 , i don`t remember. :-(
I am using keras-retinanet and have not this error.
from keras-maskrcnn.
RohitMidha23
commented on July 28, 2024
Ohh. Thanks anyway @MichaelMonashev
from keras-maskrcnn.
Related Issues (20)
- Class agnostic mask-RCNN HOT 1
- Access regression, classification and masks submodel outputs HOT 1
- which layer to extract feature image as image embedding HOT 2
- Off-by-one rounding error in OID mask resizing HOT 3
- Convert resnet50_coco_v0.2.0.h5 to tensorflow serving
- Is it possible to calculate the Dice loss ? HOT 1
- TypeError: Unexpected keyword argument passed to optimizer: learning_rate
- how to train on TPU? should I migrate keras to tf.keras?
- ModuleNotFoundError: No module named 'keras_retinanet.utils.gpu HOT 5
- AttributeError: 'tuple' object has no attribute 'layer' HOT 1
- Training error regarding retinanet HOT 2
- Config file HOT 1
- Difference between weights and snapshots in your implementation (also w/o freeze model argument) HOT 2
- AttributeError: module 'tensorflow' has no attribute 'config' HOT 1
- No mask output
- Is there any chance to have Keras_MaskRCNN in Tensorflow 2 and compatible with Keras_Retinanet ? HOT 1
- Installation fail HOT 1
- the dimension of inference result was 3 HOT 4
- in detection box[2] < box[0] HOT 3
- Weird losses behaviour HOT 17
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from keras-maskrcnn.