Code for the paper titled "Advancing instance segmentation and WBC classification in peripheral blood smear through domain adaptation: A study on PBC and the novel RV-PBS datasets" published on Elsevier's Expert Systems With Applications (ESWA) journal.
Home Page: https://www.sciencedirect.com/science/article/pii/S0957417424005268
2021-03-17 17:56:30.644710: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/home/tamal/.mujoco/mujoco200/bin:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/home/tamal/.mujoco/mujoco200/bin
2021-03-17 17:56:30.644736: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Using TensorFlow backend.
Traceback (most recent call last):
File "mask_rcnn3.py", line 267, in <module>
model_dir=MODEL_DIR)
File "/home/tamal/Research/Jimut/bloseg/Aniket_MASK_RCNN/MaskRCNN_3/Mask_RCNN/mrcnn/model.py", line 1837, in __init__
self.keras_model = self.build(mode=mode, config=config)
File "/home/tamal/Research/Jimut/bloseg/Aniket_MASK_RCNN/MaskRCNN_3/Mask_RCNN/mrcnn/model.py", line 1856, in build
shape=[None, None, config.IMAGE_SHAPE[2]], name="input_image")
File "/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/engine/input_layer.py", line 178, in Input
input_tensor=tensor)
File "/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/engine/input_layer.py", line 87, in __init__
name=self.name)
File "/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 541, in placeholder
x = tf.placeholder(dtype, shape=shape, name=name)
AttributeError: module 'tensorflow' has no attribute 'placeholder'
Labelling
[
{
"name": "bg",
"color": "#c080c0",
"attributes": []
},
{
"name": "band cell",
"color": "#33ddff",
"attributes": []
},
{
"name": "basophil",
"color": "#fa3253",
"attributes": []
},
{
"name": "blast cell",
"color": "#34d1b7",
"attributes": []
},
{
"name": "eosinophil",
"color": "#ff007c",
"attributes": []
},
{
"name": "lymphocyte",
"color": "#ff6037",
"attributes": []
},
{
"name": "monocyte",
"color": "#ddff33",
"attributes": []
},
{
"name": "neutrophil",
"color": "#b83df5",
"attributes": []
},
{
"name": "promyelocyte",
"color": "#ffcc33",
"attributes": []
},
{
"name": "myelocyte",
"color": "#24b353",
"attributes": []
},
{
"name": "metamyelocyte",
"color": "#034257",
"attributes": []
}
]
Probably create a new repository for converting slide images to its corresponding multi-class segmentation mask in using CVAT.
We can use our own segmentation model via robust data augmentation methods. We can even crop and cut the dataset with the corresponding masks to generate more and more data for segmentation.
TODO
(aniket_mask_rcnn3) tamal@tamalmj:/home/tamal/Research/Jimut/bloseg/Aniket_MASK_RCNN/MaskRCNN_3$ python3 mask_rcnn3.py
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
2021-03-17 19:37:27.586105: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-03-17 19:37:27.809481: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3699850000 Hz
2021-03-17 19:37:27.811202: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55eddfc39d70 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-03-17 19:37:27.811284: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-03-17 19:37:27.902438: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-03-17 19:37:28.314156: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:28.314918: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55eddfc24fb0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-03-17 19:37:28.314951: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce RTX 2080, Compute Capability 7.5
2021-03-17 19:37:28.325146: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:28.326027: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:02:00.0
2021-03-17 19:37:28.336059: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-03-17 19:37:28.381215: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-03-17 19:37:28.437417: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-03-17 19:37:28.455969: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-03-17 19:37:28.491466: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-03-17 19:37:28.547836: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-03-17 19:37:28.669696: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-03-17 19:37:28.670119: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:28.671616: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:28.672821: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-03-17 19:37:28.689438: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-03-17 19:37:28.698546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-03-17 19:37:28.698633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2021-03-17 19:37:28.698669: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2021-03-17 19:37:28.706308: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:28.708503: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:28.710162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7128 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:02:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/compat/v2_compat.py:68: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:541: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:66: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:4432: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:2139: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:4267: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:2239: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py:1475: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py:553: The name tf.random_shuffle is deprecated. Please use tf.random.shuffle instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/utils.py:202: The name tf.log is deprecated. Please use tf.math.log instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py:600: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:190: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:197: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:203: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2021-03-17 19:37:34.833779: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:34.834105: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:02:00.0
2021-03-17 19:37:34.834176: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-03-17 19:37:34.834194: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-03-17 19:37:34.834209: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-03-17 19:37:34.834224: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-03-17 19:37:34.834240: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-03-17 19:37:34.834255: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-03-17 19:37:34.834270: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-03-17 19:37:34.834329: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:34.834629: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:34.834895: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-03-17 19:37:34.834928: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-03-17 19:37:34.834934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2021-03-17 19:37:34.834939: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2021-03-17 19:37:34.835014: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:34.835326: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:34.835599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7128 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:02:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:207: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:216: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:223: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.
./dataset/basophil/train/images
./dataset/eosinophil/train/images
./dataset/erythroblast/train/images
./dataset/ig/train/images
./dataset/lymphocyte/train/images
./dataset/monocyte/train/images
./dataset/neutrophil/train/images
./dataset/platelet/train/images
./dataset/basophil/val/images
./dataset/eosinophil/val/images
./dataset/erythroblast/val/images
./dataset/ig/val/images
./dataset/lymphocyte/val/images
./dataset/monocyte/val/images
./dataset/neutrophil/val/images
./dataset/platelet/val/images
Train network heads
Starting at epoch 0. LR=0.001
Checkpoint Path: /home/tamal/Research/Jimut/bloseg/Aniket_MASK_RCNN/MaskRCNN_3/Mask_RCNN/logs/blood20210317T1937/mask_rcnn_blood_{epoch:04d}.h5
Selecting layers to train
fpn_c5p5 (Conv2D)
fpn_c4p4 (Conv2D)
fpn_c3p3 (Conv2D)
fpn_c2p2 (Conv2D)
fpn_p5 (Conv2D)
fpn_p2 (Conv2D)
fpn_p3 (Conv2D)
fpn_p4 (Conv2D)
In model: rpn_model
rpn_conv_shared (Conv2D)
rpn_class_raw (Conv2D)
rpn_bbox_pred (Conv2D)
mrcnn_mask_conv1 (TimeDistributed)
mrcnn_mask_bn1 (TimeDistributed)
mrcnn_mask_conv2 (TimeDistributed)
mrcnn_mask_bn2 (TimeDistributed)
mrcnn_class_conv1 (TimeDistributed)
mrcnn_class_bn1 (TimeDistributed)
mrcnn_mask_conv3 (TimeDistributed)
mrcnn_mask_bn3 (TimeDistributed)
mrcnn_class_conv2 (TimeDistributed)
mrcnn_class_bn2 (TimeDistributed)
mrcnn_mask_conv4 (TimeDistributed)
mrcnn_mask_bn4 (TimeDistributed)
mrcnn_bbox_fc (TimeDistributed)
mrcnn_mask_deconv (TimeDistributed)
mrcnn_class_logits (TimeDistributed)
mrcnn_mask (TimeDistributed)
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:1033: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:1020: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/engine/training_generator.py:49: UserWarning: Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the `keras.utils.Sequence class.
UserWarning('Using a generator with `use_multiprocessing=True`'
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/callbacks.py:1122: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/callbacks.py:1125: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.
Epoch 1/10
2021-03-17 19:38:14.326052: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-03-17 19:38:15.369728: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-03-17 19:38:20.659674: W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Not found: ./bin/ptxas not found
Relying on driver to perform ptx compilation. This message will be only logged once.
2021-03-17 19:38:24.072361: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.83GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:38:24.072436: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.83GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:38:24.139314: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:38:24.141612: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:38:24.548642: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.41GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:38:24.549016: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.41GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:38:24.947337: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.05GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:38:24.947511: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.05GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:38:25.020786: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.04GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:38:25.020836: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.04GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
1/50 [..............................] - ETA: 25:34 - loss: 17.9140 - rpn_class_loss: 1.3314 - rpn_bbox_loss: 3.6698 - mrcnn_class_loss: 12.9128 - mrcnn_bbox_loss: 0.0000e+00 - mrcnn_mask_loss: 0.0000e+0 2/50 [>.............................] - ETA: 12:41 - loss: 16.5364 - rpn_class_loss: 1.2819 - rpn_bbox_loss: 3.6159 - mrcnn_class_loss: 11.6386 - mrcnn_bbox_loss: 0.0000e+00 - mrcnn_mask_loss: 0.0000e+0 3/50 [>.............................] - ETA: 8:24 - loss: 14.5689 - rpn_class_loss: 1.2034 - rpn_bbox_loss: 3.5276 - mrcnn_class_loss: 9.8379 - mrcnn_bbox_loss: 0.0000e+00 - mrcnn_mask_loss: 0.0000e+00 50/50 [==============================] - 57s 1s/step - loss: 3.6597 - rpn_class_loss: 0.1321 - rpn_bbox_loss: 1.2943 - mrcnn_class_loss: 0.8699 - mrcnn_bbox_loss: 0.7085 - mrcnn_mask_loss: 0.6548 - val_loss: 4.8457 - val_rpn_class_loss: 0.0292 - val_rpn_bbox_loss: 3.1592 - val_mrcnn_class_loss: 0.1857 - val_mrcnn_bbox_loss: 0.7262 - val_mrcnn_mask_loss: 0.7454
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/callbacks.py:1265: The name tf.Summary is deprecated. Please use tf.compat.v1.Summary instead.
Epoch 2/10
50/50 [==============================] - 24s 482ms/step - loss: 2.1932 - rpn_class_loss: 0.0168 - rpn_bbox_loss: 0.5727 - mrcnn_class_loss: 0.2502 - mrcnn_bbox_loss: 0.6622 - mrcnn_mask_loss: 0.6913 - val_loss: 3.7600 - val_rpn_class_loss: 0.0411 - val_rpn_bbox_loss: 1.7578 - val_mrcnn_class_loss: 0.2487 - val_mrcnn_bbox_loss: 1.0156 - val_mrcnn_mask_loss: 0.6968
Epoch 3/10
50/50 [==============================] - 24s 484ms/step - loss: 2.1187 - rpn_class_loss: 0.0280 - rpn_bbox_loss: 0.7429 - mrcnn_class_loss: 0.1732 - mrcnn_bbox_loss: 0.4814 - mrcnn_mask_loss: 0.6932 - val_loss: 3.7112 - val_rpn_class_loss: 0.0285 - val_rpn_bbox_loss: 1.8525 - val_mrcnn_class_loss: 0.4423 - val_mrcnn_bbox_loss: 0.7077 - val_mrcnn_mask_loss: 0.6802
Epoch 4/10
50/50 [==============================] - 24s 481ms/step - loss: 2.0721 - rpn_class_loss: 0.0153 - rpn_bbox_loss: 0.5532 - mrcnn_class_loss: 0.2496 - mrcnn_bbox_loss: 0.6140 - mrcnn_mask_loss: 0.6401 - val_loss: 2.6142 - val_rpn_class_loss: 0.0177 - val_rpn_bbox_loss: 0.7381 - val_mrcnn_class_loss: 0.1102 - val_mrcnn_bbox_loss: 1.0741 - val_mrcnn_mask_loss: 0.6741
Epoch 5/10
50/50 [==============================] - 25s 491ms/step - loss: 1.7990 - rpn_class_loss: 0.0114 - rpn_bbox_loss: 0.2935 - mrcnn_class_loss: 0.3418 - mrcnn_bbox_loss: 0.4821 - mrcnn_mask_loss: 0.6702 - val_loss: 2.0388 - val_rpn_class_loss: 0.0165 - val_rpn_bbox_loss: 0.6126 - val_mrcnn_class_loss: 0.2847 - val_mrcnn_bbox_loss: 0.4592 - val_mrcnn_mask_loss: 0.6658
Epoch 6/10
50/50 [==============================] - 25s 490ms/step - loss: 1.7345 - rpn_class_loss: 0.0104 - rpn_bbox_loss: 0.3701 - mrcnn_class_loss: 0.2780 - mrcnn_bbox_loss: 0.4213 - mrcnn_mask_loss: 0.6547 - val_loss: 3.3186 - val_rpn_class_loss: 0.0350 - val_rpn_bbox_loss: 1.3366 - val_mrcnn_class_loss: 0.6055 - val_mrcnn_bbox_loss: 0.6839 - val_mrcnn_mask_loss: 0.6576
Epoch 7/10
50/50 [==============================] - 24s 483ms/step - loss: 1.6058 - rpn_class_loss: 0.0093 - rpn_bbox_loss: 0.3452 - mrcnn_class_loss: 0.2652 - mrcnn_bbox_loss: 0.3348 - mrcnn_mask_loss: 0.6513 - val_loss: 2.0079 - val_rpn_class_loss: 0.0183 - val_rpn_bbox_loss: 0.7288 - val_mrcnn_class_loss: 0.2194 - val_mrcnn_bbox_loss: 0.4197 - val_mrcnn_mask_loss: 0.6217
Epoch 8/10
50/50 [==============================] - 24s 481ms/step - loss: 1.6592 - rpn_class_loss: 0.0183 - rpn_bbox_loss: 0.4256 - mrcnn_class_loss: 0.2830 - mrcnn_bbox_loss: 0.2853 - mrcnn_mask_loss: 0.6471 - val_loss: 2.1766 - val_rpn_class_loss: 0.0128 - val_rpn_bbox_loss: 0.7118 - val_mrcnn_class_loss: 0.1009 - val_mrcnn_bbox_loss: 0.6933 - val_mrcnn_mask_loss: 0.6577
Epoch 9/10
50/50 [==============================] - 24s 483ms/step - loss: 1.5084 - rpn_class_loss: 0.0051 - rpn_bbox_loss: 0.3078 - mrcnn_class_loss: 0.2647 - mrcnn_bbox_loss: 0.3115 - mrcnn_mask_loss: 0.6194 - val_loss: 2.2111 - val_rpn_class_loss: 0.0076 - val_rpn_bbox_loss: 0.6412 - val_mrcnn_class_loss: 0.4763 - val_mrcnn_bbox_loss: 0.5034 - val_mrcnn_mask_loss: 0.5826
Epoch 10/10
50/50 [==============================] - 24s 483ms/step - loss: 1.6972 - rpn_class_loss: 0.0210 - rpn_bbox_loss: 0.4783 - mrcnn_class_loss: 0.2437 - mrcnn_bbox_loss: 0.3190 - mrcnn_mask_loss: 0.6352 - val_loss: 1.5141 - val_rpn_class_loss: 0.0080 - val_rpn_bbox_loss: 0.4483 - val_mrcnn_class_loss: 0.1314 - val_mrcnn_bbox_loss: 0.3691 - val_mrcnn_mask_loss: 0.5572
Traceback (most recent call last):
File "mask_rcnn3.py", line 282, in <module>
train(model, "./dataset")
File "mask_rcnn3.py", line 257, in train
hist_df_nh = pd.DataFrame(history_network_heads.history)
AttributeError: 'NoneType' object has no attribute 'history'
(aniket_mask_rcnn3) tamal@tamalmj:/home/tamal/Research/Jimut/bloseg/Aniket_MASK_RCNN/MaskRCNN_3$ python3 mask_rcnn3.py
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
2021-03-17 19:49:43.913311: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-03-17 19:49:43.941271: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3699850000 Hz
2021-03-17 19:49:43.941646: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5651829d9fc0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-03-17 19:49:43.941669: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-03-17 19:49:43.973444: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-03-17 19:49:44.133701: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:44.134086: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5651805c5c00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-03-17 19:49:44.134103: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce RTX 2080, Compute Capability 7.5
2021-03-17 19:49:44.134286: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:44.134588: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:02:00.0
2021-03-17 19:49:44.168549: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-03-17 19:49:44.197237: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-03-17 19:49:44.240665: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-03-17 19:49:44.259555: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-03-17 19:49:44.278373: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-03-17 19:49:44.289792: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-03-17 19:49:44.316497: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-03-17 19:49:44.316911: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:44.319354: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:44.321147: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-03-17 19:49:44.321412: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-03-17 19:49:44.323283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-03-17 19:49:44.323344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2021-03-17 19:49:44.323373: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2021-03-17 19:49:44.323747: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:44.325237: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:44.326523: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7138 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:02:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/compat/v2_compat.py:68: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:541: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:66: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:4432: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:2139: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:4267: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:2239: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py:1475: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py:553: The name tf.random_shuffle is deprecated. Please use tf.random.shuffle instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/utils.py:202: The name tf.log is deprecated. Please use tf.math.log instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py:600: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:190: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:197: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:203: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2021-03-17 19:49:50.674465: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:50.674789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:02:00.0
2021-03-17 19:49:50.674864: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-03-17 19:49:50.674884: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-03-17 19:49:50.674902: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-03-17 19:49:50.674921: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-03-17 19:49:50.674939: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-03-17 19:49:50.674957: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-03-17 19:49:50.674974: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-03-17 19:49:50.675034: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:50.675333: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:50.675598: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-03-17 19:49:50.675631: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-03-17 19:49:50.675637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2021-03-17 19:49:50.675642: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2021-03-17 19:49:50.675706: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:50.676012: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:50.676299: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7138 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:02:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:207: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:216: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:223: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.
./dataset/basophil/train/images
./dataset/eosinophil/train/images
./dataset/erythroblast/train/images
./dataset/ig/train/images
./dataset/lymphocyte/train/images
./dataset/monocyte/train/images
./dataset/neutrophil/train/images
./dataset/platelet/train/images
./dataset/basophil/val/images
./dataset/eosinophil/val/images
./dataset/erythroblast/val/images
./dataset/ig/val/images
./dataset/lymphocyte/val/images
./dataset/monocyte/val/images
./dataset/neutrophil/val/images
./dataset/platelet/val/images
Train network heads
Starting at epoch 0. LR=0.001
Checkpoint Path: /home/tamal/Research/Jimut/bloseg/Aniket_MASK_RCNN/MaskRCNN_3/Mask_RCNN/logs/blood20210317T1949/mask_rcnn_blood_{epoch:04d}.h5
Selecting layers to train
fpn_c5p5 (Conv2D)
fpn_c4p4 (Conv2D)
fpn_c3p3 (Conv2D)
fpn_c2p2 (Conv2D)
fpn_p5 (Conv2D)
fpn_p2 (Conv2D)
fpn_p3 (Conv2D)
fpn_p4 (Conv2D)
In model: rpn_model
rpn_conv_shared (Conv2D)
rpn_class_raw (Conv2D)
rpn_bbox_pred (Conv2D)
mrcnn_mask_conv1 (TimeDistributed)
mrcnn_mask_bn1 (TimeDistributed)
mrcnn_mask_conv2 (TimeDistributed)
mrcnn_mask_bn2 (TimeDistributed)
mrcnn_class_conv1 (TimeDistributed)
mrcnn_class_bn1 (TimeDistributed)
mrcnn_mask_conv3 (TimeDistributed)
mrcnn_mask_bn3 (TimeDistributed)
mrcnn_class_conv2 (TimeDistributed)
mrcnn_class_bn2 (TimeDistributed)
mrcnn_mask_conv4 (TimeDistributed)
mrcnn_mask_bn4 (TimeDistributed)
mrcnn_bbox_fc (TimeDistributed)
mrcnn_mask_deconv (TimeDistributed)
mrcnn_class_logits (TimeDistributed)
mrcnn_mask (TimeDistributed)
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:1033: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:1020: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/engine/training_generator.py:49: UserWarning: Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the `keras.utils.Sequence class.
UserWarning('Using a generator with `use_multiprocessing=True`'
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/callbacks.py:1122: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/callbacks.py:1125: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.
Epoch 1/4
2021-03-17 19:50:18.633214: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-03-17 19:50:18.764498: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-03-17 19:50:19.431184: W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Not found: ./bin/ptxas not found
Relying on driver to perform ptx compilation. This message will be only logged once.
2021-03-17 19:50:22.532819: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.83GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:22.532862: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.83GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:22.533247: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.05GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:22.533287: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.05GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:22.608080: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.04GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:22.608192: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.04GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:22.612358: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:22.612779: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:23.048745: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.41GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:23.048780: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.41GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:23.400585: I tensorflow/stream_executor/cuda/cuda_driver.cc:831] failed to allocate 993.83M (1042110720 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-03-17 19:50:23.401123: I tensorflow/stream_executor/cuda/cuda_driver.cc:831] failed to allocate 894.45M (937899776 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
50/50 [==============================] - 50s 997ms/step - loss: 2.8902 - rpn_class_loss: 0.3050 - rpn_bbox_loss: 0.9136 - mrcnn_class_loss: 0.3932 - mrcnn_bbox_loss: 0.7135 - mrcnn_mask_loss: 0.5649 - val_loss: 4.3391 - val_rpn_class_loss: 0.0424 - val_rpn_bbox_loss: 1.3799 - val_mrcnn_class_loss: 0.6488 - val_mrcnn_bbox_loss: 1.5979 - val_mrcnn_mask_loss: 0.6701
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/callbacks.py:1265: The name tf.Summary is deprecated. Please use tf.compat.v1.Summary instead.
Epoch 2/4
50/50 [==============================] - 24s 476ms/step - loss: 2.5034 - rpn_class_loss: 0.0294 - rpn_bbox_loss: 0.7938 - mrcnn_class_loss: 0.3023 - mrcnn_bbox_loss: 0.6994 - mrcnn_mask_loss: 0.6785 - val_loss: 3.2076 - val_rpn_class_loss: 0.0212 - val_rpn_bbox_loss: 1.0725 - val_mrcnn_class_loss: 0.3652 - val_mrcnn_bbox_loss: 1.0609 - val_mrcnn_mask_loss: 0.6879
Epoch 3/4
50/50 [==============================] - 24s 478ms/step - loss: 2.0304 - rpn_class_loss: 0.0228 - rpn_bbox_loss: 0.4881 - mrcnn_class_loss: 0.2915 - mrcnn_bbox_loss: 0.5484 - mrcnn_mask_loss: 0.6796 - val_loss: 4.0582 - val_rpn_class_loss: 0.0321 - val_rpn_bbox_loss: 2.6589 - val_mrcnn_class_loss: 0.1285 - val_mrcnn_bbox_loss: 0.5701 - val_mrcnn_mask_loss: 0.6687
Epoch 4/4
50/50 [==============================] - 24s 479ms/step - loss: 2.1744 - rpn_class_loss: 0.0287 - rpn_bbox_loss: 0.7892 - mrcnn_class_loss: 0.2251 - mrcnn_bbox_loss: 0.4589 - mrcnn_mask_loss: 0.6725 - val_loss: 2.6046 - val_rpn_class_loss: 0.0262 - val_rpn_bbox_loss: 1.4513 - val_mrcnn_class_loss: 0.0517 - val_mrcnn_bbox_loss: 0.4163 - val_mrcnn_mask_loss: 0.6590
Train all layers
Starting at epoch 4. LR=0.001
Checkpoint Path: /home/tamal/Research/Jimut/bloseg/Aniket_MASK_RCNN/MaskRCNN_3/Mask_RCNN/logs/blood20210317T1949/mask_rcnn_blood_{epoch:04d}.h5
Selecting layers to train
conv1 (Conv2D)
bn_conv1 (BatchNorm)
res2a_branch2a (Conv2D)
bn2a_branch2a (BatchNorm)
res2a_branch2b (Conv2D)
bn2a_branch2b (BatchNorm)
res2a_branch2c (Conv2D)
res2a_branch1 (Conv2D)
bn2a_branch2c (BatchNorm)
bn2a_branch1 (BatchNorm)
res2b_branch2a (Conv2D)
bn2b_branch2a (BatchNorm)
res2b_branch2b (Conv2D)
bn2b_branch2b (BatchNorm)
res2b_branch2c (Conv2D)
bn2b_branch2c (BatchNorm)
res2c_branch2a (Conv2D)
bn2c_branch2a (BatchNorm)
res2c_branch2b (Conv2D)
bn2c_branch2b (BatchNorm)
res2c_branch2c (Conv2D)
bn2c_branch2c (BatchNorm)
res3a_branch2a (Conv2D)
bn3a_branch2a (BatchNorm)
res3a_branch2b (Conv2D)
bn3a_branch2b (BatchNorm)
res3a_branch2c (Conv2D)
res3a_branch1 (Conv2D)
bn3a_branch2c (BatchNorm)
bn3a_branch1 (BatchNorm)
res3b_branch2a (Conv2D)
bn3b_branch2a (BatchNorm)
res3b_branch2b (Conv2D)
bn3b_branch2b (BatchNorm)
res3b_branch2c (Conv2D)
bn3b_branch2c (BatchNorm)
res3c_branch2a (Conv2D)
bn3c_branch2a (BatchNorm)
res3c_branch2b (Conv2D)
bn3c_branch2b (BatchNorm)
res3c_branch2c (Conv2D)
bn3c_branch2c (BatchNorm)
res3d_branch2a (Conv2D)
bn3d_branch2a (BatchNorm)
res3d_branch2b (Conv2D)
bn3d_branch2b (BatchNorm)
res3d_branch2c (Conv2D)
bn3d_branch2c (BatchNorm)
res4a_branch2a (Conv2D)
bn4a_branch2a (BatchNorm)
res4a_branch2b (Conv2D)
bn4a_branch2b (BatchNorm)
res4a_branch2c (Conv2D)
res4a_branch1 (Conv2D)
bn4a_branch2c (BatchNorm)
bn4a_branch1 (BatchNorm)
res4b_branch2a (Conv2D)
bn4b_branch2a (BatchNorm)
res4b_branch2b (Conv2D)
bn4b_branch2b (BatchNorm)
res4b_branch2c (Conv2D)
bn4b_branch2c (BatchNorm)
res4c_branch2a (Conv2D)
bn4c_branch2a (BatchNorm)
res4c_branch2b (Conv2D)
bn4c_branch2b (BatchNorm)
res4c_branch2c (Conv2D)
bn4c_branch2c (BatchNorm)
res4d_branch2a (Conv2D)
bn4d_branch2a (BatchNorm)
res4d_branch2b (Conv2D)
bn4d_branch2b (BatchNorm)
res4d_branch2c (Conv2D)
bn4d_branch2c (BatchNorm)
res4e_branch2a (Conv2D)
bn4e_branch2a (BatchNorm)
res4e_branch2b (Conv2D)
bn4e_branch2b (BatchNorm)
res4e_branch2c (Conv2D)
bn4e_branch2c (BatchNorm)
res4f_branch2a (Conv2D)
bn4f_branch2a (BatchNorm)
res4f_branch2b (Conv2D)
bn4f_branch2b (BatchNorm)
res4f_branch2c (Conv2D)
bn4f_branch2c (BatchNorm)
res4g_branch2a (Conv2D)
bn4g_branch2a (BatchNorm)
res4g_branch2b (Conv2D)
bn4g_branch2b (BatchNorm)
res4g_branch2c (Conv2D)
bn4g_branch2c (BatchNorm)
res4h_branch2a (Conv2D)
bn4h_branch2a (BatchNorm)
res4h_branch2b (Conv2D)
bn4h_branch2b (BatchNorm)
res4h_branch2c (Conv2D)
bn4h_branch2c (BatchNorm)
res4i_branch2a (Conv2D)
bn4i_branch2a (BatchNorm)
res4i_branch2b (Conv2D)
bn4i_branch2b (BatchNorm)
res4i_branch2c (Conv2D)
bn4i_branch2c (BatchNorm)
res4j_branch2a (Conv2D)
bn4j_branch2a (BatchNorm)
res4j_branch2b (Conv2D)
bn4j_branch2b (BatchNorm)
res4j_branch2c (Conv2D)
bn4j_branch2c (BatchNorm)
res4k_branch2a (Conv2D)
bn4k_branch2a (BatchNorm)
res4k_branch2b (Conv2D)
bn4k_branch2b (BatchNorm)
res4k_branch2c (Conv2D)
bn4k_branch2c (BatchNorm)
res4l_branch2a (Conv2D)
bn4l_branch2a (BatchNorm)
res4l_branch2b (Conv2D)
bn4l_branch2b (BatchNorm)
res4l_branch2c (Conv2D)
bn4l_branch2c (BatchNorm)
res4m_branch2a (Conv2D)
bn4m_branch2a (BatchNorm)
res4m_branch2b (Conv2D)
bn4m_branch2b (BatchNorm)
res4m_branch2c (Conv2D)
bn4m_branch2c (BatchNorm)
res4n_branch2a (Conv2D)
bn4n_branch2a (BatchNorm)
res4n_branch2b (Conv2D)
bn4n_branch2b (BatchNorm)
res4n_branch2c (Conv2D)
bn4n_branch2c (BatchNorm)
res4o_branch2a (Conv2D)
bn4o_branch2a (BatchNorm)
res4o_branch2b (Conv2D)
bn4o_branch2b (BatchNorm)
res4o_branch2c (Conv2D)
bn4o_branch2c (BatchNorm)
res4p_branch2a (Conv2D)
bn4p_branch2a (BatchNorm)
res4p_branch2b (Conv2D)
bn4p_branch2b (BatchNorm)
res4p_branch2c (Conv2D)
bn4p_branch2c (BatchNorm)
res4q_branch2a (Conv2D)
bn4q_branch2a (BatchNorm)
res4q_branch2b (Conv2D)
bn4q_branch2b (BatchNorm)
res4q_branch2c (Conv2D)
bn4q_branch2c (BatchNorm)
res4r_branch2a (Conv2D)
bn4r_branch2a (BatchNorm)
res4r_branch2b (Conv2D)
bn4r_branch2b (BatchNorm)
res4r_branch2c (Conv2D)
bn4r_branch2c (BatchNorm)
res4s_branch2a (Conv2D)
bn4s_branch2a (BatchNorm)
res4s_branch2b (Conv2D)
bn4s_branch2b (BatchNorm)
res4s_branch2c (Conv2D)
bn4s_branch2c (BatchNorm)
res4t_branch2a (Conv2D)
bn4t_branch2a (BatchNorm)
res4t_branch2b (Conv2D)
bn4t_branch2b (BatchNorm)
res4t_branch2c (Conv2D)
bn4t_branch2c (BatchNorm)
res4u_branch2a (Conv2D)
bn4u_branch2a (BatchNorm)
res4u_branch2b (Conv2D)
bn4u_branch2b (BatchNorm)
res4u_branch2c (Conv2D)
bn4u_branch2c (BatchNorm)
res4v_branch2a (Conv2D)
bn4v_branch2a (BatchNorm)
res4v_branch2b (Conv2D)
bn4v_branch2b (BatchNorm)
res4v_branch2c (Conv2D)
bn4v_branch2c (BatchNorm)
res4w_branch2a (Conv2D)
bn4w_branch2a (BatchNorm)
res4w_branch2b (Conv2D)
bn4w_branch2b (BatchNorm)
res4w_branch2c (Conv2D)
bn4w_branch2c (BatchNorm)
res5a_branch2a (Conv2D)
bn5a_branch2a (BatchNorm)
res5a_branch2b (Conv2D)
bn5a_branch2b (BatchNorm)
res5a_branch2c (Conv2D)
res5a_branch1 (Conv2D)
bn5a_branch2c (BatchNorm)
bn5a_branch1 (BatchNorm)
res5b_branch2a (Conv2D)
bn5b_branch2a (BatchNorm)
res5b_branch2b (Conv2D)
bn5b_branch2b (BatchNorm)
res5b_branch2c (Conv2D)
bn5b_branch2c (BatchNorm)
res5c_branch2a (Conv2D)
bn5c_branch2a (BatchNorm)
res5c_branch2b (Conv2D)
bn5c_branch2b (BatchNorm)
res5c_branch2c (Conv2D)
bn5c_branch2c (BatchNorm)
fpn_c5p5 (Conv2D)
fpn_c4p4 (Conv2D)
fpn_c3p3 (Conv2D)
fpn_c2p2 (Conv2D)
fpn_p5 (Conv2D)
fpn_p2 (Conv2D)
fpn_p3 (Conv2D)
fpn_p4 (Conv2D)
In model: rpn_model
rpn_conv_shared (Conv2D)
rpn_class_raw (Conv2D)
rpn_bbox_pred (Conv2D)
mrcnn_mask_conv1 (TimeDistributed)
mrcnn_mask_bn1 (TimeDistributed)
mrcnn_mask_conv2 (TimeDistributed)
mrcnn_mask_bn2 (TimeDistributed)
mrcnn_class_conv1 (TimeDistributed)
mrcnn_class_bn1 (TimeDistributed)
mrcnn_mask_conv3 (TimeDistributed)
mrcnn_mask_bn3 (TimeDistributed)
mrcnn_class_conv2 (TimeDistributed)
mrcnn_class_bn2 (TimeDistributed)
mrcnn_mask_conv4 (TimeDistributed)
mrcnn_mask_bn4 (TimeDistributed)
mrcnn_bbox_fc (TimeDistributed)
mrcnn_mask_deconv (TimeDistributed)
mrcnn_class_logits (TimeDistributed)
mrcnn_mask (TimeDistributed)
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/engine/training_generator.py:49: UserWarning: Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the `keras.utils.Sequence class.
UserWarning('Using a generator with `use_multiprocessing=True`'
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py:758: The name tf.sets.set_intersection is deprecated. Please use tf.sets.intersection instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py:760: The name tf.sparse_tensor_to_dense is deprecated. Please use tf.sparse.to_dense instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py:772: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
Re-starting from epoch 4
./dataset/basophil/val/images
./dataset/eosinophil/val/images
./dataset/erythroblast/val/images
./dataset/ig/val/images
./dataset/lymphocyte/val/images
./dataset/monocyte/val/images
./dataset/neutrophil/val/images
./dataset/platelet/val/images
original_image shape: (1024, 1024, 3) min: 0.00000 max: 255.00000 uint8
image_meta shape: (21,) min: 1.00000 max: 1024.00000 float64
gt_class_id shape: (1,) min: 1.00000 max: 1.00000 int32
gt_bbox shape: (1, 4) min: 357.00000 max: 673.00000 int32
gt_mask shape: (1024, 1024, 1) min: 0.00000 max: 1.00000 bool
Processing 1 images
image shape: (1024, 1024, 3) min: 0.00000 max: 255.00000 uint8
molded_images shape: (1, 1024, 1024, 3) min: -123.70000 max: 137.20000 float64
image_metas shape: (1, 21) min: 0.00000 max: 1024.00000 int64
anchors shape: (1, 261888, 4) min: -0.35390 max: 1.29134 float32
True: BA
Prediction
['EO'] ===> 0.9152153
['EO'] ===> 0.6645939
['EO'] ===> 0.64412767
['EO'] ===> 0.63139707
['EO'] ===> 0.57328814
Epoch 1/10
2021-03-17 19:23:00.590850: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-03-17 19:23:02.466831: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-03-17 19:23:07.289605: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2021-03-17 19:23:07.294961: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Traceback (most recent call last):
File "mask_rcnn3.py", line 277, in
train(model, "./dataset")
File "mask_rcnn3.py", line 250, in train
layers='heads')
File "/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py", line 2374, in train
File "/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/engine/training.py", line 1658, in fit_generator
initial_epoch=initial_epoch)
File "/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/engine/training_generator.py", line 215, in fit_generator
class_weight=class_weight)
File "/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/engine/training.py", line 1449, in train_on_batch
outputs = self.train_function(ins)
File "/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 2979, in call
return self._call(inputs)
File "/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 2937, in _call
fetched = self._callable_fn(*array_vals)
File "/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1472, in call
run_metadata_ptr)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node conv1/convolution}}]]
[[proposal_targets/strided_slice_53/_2887]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node conv1/convolution}}]]
0 successful operations.
0 derived errors ignored.
Neutrophil done
Instead of storing dataset in Git, we can store it somewhere else and use DVC to manage it. Probably a better idea
@heraldofsolace Look into this matter please...
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.