mlcv-ime / faster-rcnn-graphics Goto Github PK

Symbol detection in online handwritten graphics using Faster R-CNN

Python 93.02% Shell 6.98%

faster-r-cnn flowchart symbol-detection tensorflow handwriting-recognition graphics mathematical-expressions object-detection

faster-rcnn-graphics's Introduction

Symbol detection in online handwritten graphics using Faster R-CNN

faster-rcnn-graphics has been imported to ImageU. Future updates will only be done on that project

This repository contains the implementation of the models described in the paper "Symbol detection in online handwritten graphics using Faster R-CNN". A model is a Faster R-CNN network that takes an image of a handwritten graphic (flowchart or mathematical expression) as input and predicts the bounding box coordinates of the symbols that compose the graphic. The models are implemented using a fork of the the Tensorflow Object Detection API.

Symbol detection in flowchart

Symbol detetion in mathematical expression

Citing this work

In case you use this work, please consider citing:

@inproceedings{frankdas:2018,
  title={Symbol detection in online handwritten graphics using Faster R-CNN},
  author={Frank Julca-Aguilar and Nina Hirata},
  booktitle={13th IAPR International Workshop on Document Analysis Systems (DAS)},
  year={2018}
 }

Installation
Evaluating the models
Training new models

Installation

Clone the repository (with --recursive)

git clone --recursive https://github.com/vision-ime/faster-rcnn-graphics.git

The recursive option is necessary to download the fork version of the Tensorflow Object Detection API used in our experimentation.

Follow the Tensorflow Object Detection API installation instructions to set up the API, which was cloned in the tf-models folder (the tf-models folder corresponds to the models folder described in the API installation instructions).

Evaluating the models

Download the datasets. In the directory where you cloned this repository do:

./download_datasets.sh

The datasets will be saved in the datasets folder. Each dataset consist of Tensorflow's .record files, images of handwritten graphics, and xml metadata for each image. As described in the paper, the datasets were using the CROHME-2016 and flowcharts datasets.

Download a model. Models can be download from: http://www.vision.ime.usp.br/~frank.aguilar/graphics/models/

For example, to download the model for symbol detection in flowcharts, trained with inception V2:

wget http://www.vision.ime.usp.br/~frank.aguilar/graphics/models/flowcharts/flowcharts_inceptionv2.tar.gzip

In order better to organize the different files, we can
save the model in the corresponding flowchart folder.

mkdir models/flowcharts/inceptionv2/trained
mv flowcharts_inceptionv2.tar.gzip models/flowcharts/inceptionv2/trained/
cd models/flowcharts/inceptionv2/trained
tar -xf flowcharts_inceptionv2.tar.gzip

Execute the evaluation script. In the folder in which you cloned this work, to evaluate the model downloaded in step 2, you can do

python tf-models/research/object_detection/eval.py \
    --logtostderr \
    --pipeline_config_path=models/flowcharts/inceptionv2/pipeline.config \
    --checkpoint_dir=models/flowcharts/inceptionv2/trained \
    --eval_dir=models/flowcharts/inceptionv2/eval \ 
    --gpudev=0 \
    --run_once=True

The parameter gpudev indicates the GPU device that would be used to evaluate the model. A value -1 can be used to run over CPU. The rest of the parameters are defined as in the Object Detection API (here).

Training new models

New models can be trained using

python tf-models/research/object_detection/train.py \
--logtostderr \
--pipeline_config_path=models/math/inceptionv2/pipeline.config \
--train_dir=models/math/inceptionv2/new_trained \ 
--gpudev=0 &

As in the case of evaluation, the parameters are defined
here.

faster-rcnn-graphics's People

Contributors

Stargazers

Watchers

Forkers

fendaq kastland bekerov dmansilla07 yue1harriet1 sanyog42 tanaykarve cdchushig dfuchss-research sabeehalam jerichomcleod

faster-rcnn-graphics's Issues

The environment configuration is invalid, I need help

Some documents guiding environment installation have expired.
Can you provide the download method of your original environment after packaging

tensorflow.python.framework.errors_impl.NotFoundError: datasets/flowcharts/flowcharts768_val.record; No such file or directory

Upon executing
python3 tf-models/research/object_detection/eval.py --pipeline_config_path=models/flowcharts/inceptionv2/pipeline.config --checkpoint_dir=models/flowcharts/inceptionv2/trained --eval_dir=models/flowcharts/inceptionv2/eval --gpudev=-1 --run_once=True
I get the following error
INFO:tensorflow:Scale of 0 disables regularizer. INFO:tensorflow:Scale of 0 disables regularizer. INFO:tensorflow:depth of additional conv before box predictor: 0 WARNING:tensorflow:From /home/viral/Code/Python/MachineLearning/faster-rcnn-graphics/tf-models/research/object_detection/core/box_predictor.py:380: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead 2018-10-08 21:56:47.414693: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA INFO:tensorflow:Restoring parameters from models/flowcharts/inceptionv2/trained/model.ckpt-25000 INFO:tensorflow:Restoring parameters from models/flowcharts/inceptionv2/trained/model.ckpt-25000 INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.NotFoundError'>, datasets/flowcharts/flowcharts768_val.record; No such file or directory [[Node: parallel_read/ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](parallel_read/TFRecordReaderV2, parallel_read/filenames)]] INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.NotFoundError'>, datasets/flowcharts/flowcharts768_val.record; No such file or directory [[Node: parallel_read/ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](parallel_read/TFRecordReaderV2, parallel_read/filenames)]] WARNING:root:The following classes have no ground truth examples: [1 2 3 4 5 6 7] /home/viral/Code/Python/MachineLearning/faster-rcnn-graphics/tf-models/research/object_detection/utils/metrics.py:144: RuntimeWarning: invalid value encountered in true_divide num_images_correctly_detected_per_class / num_gt_imgs_per_class) /home/viral/Code/Python/MachineLearning/faster-rcnn-graphics/tf-models/research/object_detection/utils/object_detection_evaluation.py:612: RuntimeWarning: Mean of empty slice mean_ap = np.nanmean(self.average_precision_per_class) /home/viral/Code/Python/MachineLearning/faster-rcnn-graphics/tf-models/research/object_detection/utils/object_detection_evaluation.py:613: RuntimeWarning: Mean of empty slice mean_corloc = np.nanmean(self.corloc_per_class) Traceback (most recent call last): File "tf-models/research/object_detection/eval.py", line 158, in <module> tf.app.run() File "/home/viral/.local/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run _sys.exit(main(argv)) File "tf-models/research/object_detection/eval.py", line 145, in main FLAGS.checkpoint_dir, FLAGS.eval_dir) File "/home/viral/Code/Python/MachineLearning/faster-rcnn-graphics/tf-models/research/object_detection/evaluator.py", line 210, in evaluate save_graph_dir=(eval_dir if eval_config.save_graph else '')) File "/home/viral/Code/Python/MachineLearning/faster-rcnn-graphics/tf-models/research/object_detection/eval_util.py", line 381, in repeated_checkpoint_run save_graph_dir) File "/home/viral/Code/Python/MachineLearning/faster-rcnn-graphics/tf-models/research/object_detection/eval_util.py", line 286, in _run_checkpoint_once global_step = tf.train.global_step(sess, tf.train.get_global_step()) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/viral/.local/lib/python3.6/site-packages/tensorflow/contrib/slim/python/slim/queues.py", line 70, in QueueRunners coord.join(threads, stop_grace_period_secs=120) File "/home/viral/.local/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 389, in join six.reraise(*self._exc_info_to_raise) File "/home/viral/.local/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/home/viral/.local/lib/python3.6/site-packages/tensorflow/python/training/queue_runner_impl.py", line 252, in _run enqueue_callable() File "/home/viral/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1205, in _single_operation_run self._call_tf_sessionrun(None, {}, [], target_list, None) File "/home/viral/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.NotFoundError: datasets/flowcharts/flowcharts768_val.record; No such file or directory [[Node: parallel_read/ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](parallel_read/TFRecordReaderV2, parallel_read/filenames)]]

Assertion error - checkpoint_dir is missing

Upon executing the command
python tf-models/research/object_detection/eval.py \ --logtostderr \ --pipeline_config_path=models/flowcharts/inceptionv2/pipeline.config \ --checkpoint_dir=models/flowcharts/inceptionv2/trained \ --eval_dir=models/flowcharts/inceptionv2/eval \ --gpudev=0 \ --run_once=True
I get the following error
File "eval.py", line 93, in main assert FLAGS.checkpoint_dir, 'checkpoint_diris missing.' AssertionError:checkpoint_dir is missing.
I followed through the setup as mentioned in the readme and everything else works perfectly fine, even executing
python object_detection/builders/model_builder_test.py works perfectly fine.

Cannot import name 'string_int_label_map_pb2'

When I try to run your project it seems i get this what is wrong with it?
By the way I can't find this file in object_detections.protos
Traceback (most recent call last): File "tf-models/research/object_detection/eval.py", line 56, in <module> from object_detection import evaluator File "/home/hackerghost/flowchat/faster-rcnn-graphics/tf-models/research/object_detection/evaluator.py", line 24, in <module> from object_detection import eval_util File "/home/hackerghost/flowchat/faster-rcnn-graphics/tf-models/research/object_detection/eval_util.py", line 27, in <module> from object_detection.utils import label_map_util File "/home/hackerghost/flowchat/faster-rcnn-graphics/tf-models/research/object_detection/utils/label_map_util.py", line 22, in <module> from object_detection.protos import string_int_label_map_pb2 ImportError: cannot import name 'string_int_label_map_pb2'

Unable to create .pb file from .ckpt and .meta files

Hi,

I have completed all installation related steps and tried to create .pb file from checkpoint files using
python3 export_inference_graph.py --input_type image_tensor --pipeline_config_path inceptionv2/pipeline.config --trained_checkpoint_prefix inceptionv2/model.ckpt --output_directory trained-inference-graphs/output_inference_graph_v1.pb
but getting error :
_2019-12-26 18:04:24.562143: W tensorflow/core/util/tensor_slice_reader.cc:95] Could not open inceptionv2/model.ckpt: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
2019-12-26 18:04:24.562163: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: Unable to open table file inceptionv2/model.ckpt: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
Traceback (most recent call last):
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/fc_venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1323, in _do_call
return fn(*args)
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/fc_venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1302, in _run_fn
status, run_metadata)
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/fc_venv/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.DataLossError: Unable to open table file inceptionv2/model.ckpt: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
[[Node: save/RestoreV2_356 = RestoreV2[dtypes=[DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2_356/tensor_names, save/RestoreV2_356/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "export_inference_graph.py", line 119, in
tf.app.run()
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/fc_venv/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "export_inference_graph.py", line 115, in main
FLAGS.output_directory, input_shape)
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/LiveFlowchart_Symbol_detection/faster-rcnn-graphics/tf-models/research/object_detection/exporter.py", line 427, in export_inference_graph
input_shape, optimize_graph, output_collection_name)
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/LiveFlowchart_Symbol_detection/faster-rcnn-graphics/tf-models/research/object_detection/exporter.py", line 375, in _export_inference_graph
trained_checkpoint_prefix=checkpoint_to_use)
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/LiveFlowchart_Symbol_detection/faster-rcnn-graphics/tf-models/research/object_detection/exporter.py", line 320, in _write_graph_and_checkpoint
saver.restore(sess, trained_checkpoint_prefix)
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/fc_venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1666, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/fc_venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/fc_venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/fc_venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/fc_venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.DataLossError: Unable to open table file inceptionv2/model.ckpt: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
[[Node: save/RestoreV2_356 = RestoreV2[dtypes=[DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2_356/tensor_names, save/RestoreV2_356/shape_and_slices)]]

Caused by op 'save/RestoreV2_356', defined at:
File "export_inference_graph.py", line 119, in
tf.app.run()
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/fc_venv/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "export_inference_graph.py", line 115, in main
FLAGS.output_directory, input_shape)
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/LiveFlowchart_Symbol_detection/faster-rcnn-graphics/tf-models/research/object_detection/exporter.py", line 427, in export_inference_graph
input_shape, optimize_graph, output_collection_name)
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/LiveFlowchart_Symbol_detection/faster-rcnn-graphics/tf-models/research/object_detection/exporter.py", line 375, in _export_inference_graph
trained_checkpoint_prefix=checkpoint_to_use)
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/LiveFlowchart_Symbol_detection/faster-rcnn-graphics/tf-models/research/object_detection/exporter.py", line 316, in _write_graph_and_checkpoint
tf.import_graph_def(inference_graph_def, name='')
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/fc_venv/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 313, in import_graph_def
op_def=op_def)
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/fc_venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
File "/home/soumitra/PycharmProjects/LiveFlowchartProject/fc_venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

DataLossError (see above for traceback): Unable to open table file inceptionv2/model.ckpt: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
[[Node: save/RestoreV2_356 = RestoreV2[dtypes=[DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](arg_save/Const_0_0, save/RestoreV2_356/tensor_names, save/RestoreV2_356/shape_and_slices)]]

I googled the error and I suspect it can be due to tensorflow version mismatch between the version used to train the model and the version used for freezing the graph or creating .pb file.

Can you please share requirements.txt file?
I almost tried with all tensorflow versions but no respite.
Also, the jupyter notebook present tf-models/research/object_detection/ is also shows error -

unreadable notebook jupyter AttributeError('metadata',)

Thanks in advance.

Dataset links not working

Hi,
I was trying to run your model but the dataset links are not valid. Can you please share correct url for flowchart dataset which you used? I tried downloading the dataset from here but the data contents seem different.

Thanks,
Rohit

mlcv-ime / faster-rcnn-graphics Goto Github PK

faster-rcnn-graphics's Introduction

Symbol detection in online handwritten graphics using Faster R-CNN

faster-rcnn-graphics has been imported to ImageU. Future updates will only be done on that project

Citing this work

Contents

Installation

Evaluating the models

Training new models

faster-rcnn-graphics's People

Contributors

Stargazers

Watchers

Forkers

faster-rcnn-graphics's Issues

Recommend Projects

Recommend Topics

Recommend Org