Comments (10)
ChWick
commented on June 18, 2024
Some calamari models in the repository are not yet trained with the moving average (older calamari version), however this has no effect on the prediction.
The deprecated warnings will be 'fixed' in a future update, to fix "TensorFlow binary was not compiled to use: AVX2 FMA" you need to compile Tensorflow for your machine.
from calamari.
alwc
commented on June 18, 2024
@ChWick I've moved all the test .png and .gt.txt into test-calamari and ran the following code:
calamari-predict --checkpoint calamari_models/default/ModernEnglish.ckpt --files /test-calamari/*.png
Next, I ran
calamari-eval --gt data/test-calamari/*.gt.txt
and here are the result:
Loading Dataset: 100% 9730/9730 [00:00<00:00, 40378.65it/s]
Text Preprocessing: 100% 9730/9730 [00:01<00:00, 6694.55it/s]
Loading Dataset: 100% 9730/9730 [00:00<00:00, 43940.55it/s]
Text Preprocessing: 100% 9730/9730 [00:01<00:00, 7067.60it/s]
Evaluation: 100% 9730/9730 [00:03<00:00, 2773.32it/s]
Evaluation result
=================
Got mean normalized label error rate of 0.43% (1957 errs, 459429 total chars, 1973 sync errs)
GT PRED COUNT PERCENT
{ } {} 76 3.85%
{} { } 55 2.79%
{.} {,} 48 2.43%
{,} {.} 42 2.13%
{'} {} 31 1.57%
{e} {c} 29 1.47%
{O} {0} 26 1.32%
{i} {l} 19 0.96%
{1} {l} 17 0.86%
{} {.} 15 0.76%
The remaining but hidden errors make up 81.86%
Should I get normalized label error rate close to 0.155% (as indicated in the paper) if I'm using the ModernEnglish.ckpt pretrained weights? Thanks.
from calamari.
ChWick
commented on June 18, 2024
@alwc I updated both the model and calamari to the current version (0.1.8!). Note that you need to specify the --checkpoint now also for calamari-eval to get the correct preprocessing of the gt files (e.g. resolve roman digits, quotes, ...).
To check for all errors during the evaluation you should specify --n_confusions -1. Possible there exist a few lines that are 'emtpy' as image but their gt does exist. (We ignored those lines as wrong GT).
from calamari.
alwc
commented on June 18, 2024
@ChWick Thanks for your help! So I updated calamari_ocr to 0.1.8 and downloaded the new pretrained models. Here were the steps I did afterward:
$: calamari-predict --checkpoint calamari_models/default/ModernEnglish.ckpt --files data/test-calamari/*.png
Found 9730 files in the dataset
2018-08-07 02:54:08.754899: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
Using CUDNN compatible LSTM backend on CPU
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/rnn.py:417: calling reverse_sequence (from tensorflow.python.ops.array_ops) with seq_dim is deprecated and will be removed in a future version.
Instructions for updating:
seq_dim is deprecated, use seq_axis instead
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py:432: calling reverse_sequence (from tensorflow.python.ops.array_ops) with batch_dim is deprecated and will be removed in a future version.
Instructions for updating:
batch_dim is deprecated, use batch_axis instead
Loading Dataset: 100%|##########################################################################################################################| 9730/9730 [00:13<00:00, 737.25it/s]
Data Preprocessing: 100%|########################################################################################################################| 9730/9730 [02:27<00:00, 65.76it/s]
Prediction: 100%|################################################################################################################################| 9730/9730 [29:41<00:00, 5.46it/s]
Prediction of 1 models took 1942.9190566539764s
All files written
Next, I ran the evaluation:
$: calamari-eval --gt data/test-calamari/*.gt.txt --checkpoint calamari_models/default/ModernEnglish.ckpt
Resolving files
Loading Dataset: 100%|########################################################################################################################| 9730/9730 [00:00<00:00, 40278.74it/s]
Text Preprocessing: 100%|#####################################################################################################################| 9730/9730 [00:00<00:00, 13625.54it/s]
Loading Dataset: 100%|########################################################################################################################| 9730/9730 [00:00<00:00, 32110.62it/s]
Text Preprocessing: 100%|#####################################################################################################################| 9730/9730 [00:00<00:00, 13351.66it/s]
Evaluation: 100%|##############################################################################################################################| 9730/9730 [00:02<00:00, 4504.07it/s]
Evaluation result
=================
Got mean normalized label error rate of 0.23% (1078 errs, 459464 total chars, 1082 sync errs)
GT PRED COUNT PERCENT
{,} {.} 33 3.05%
{} { } 32 2.96%
{ } {} 31 2.87%
{.} {,} 18 1.66%
{i} {l} 11 1.02%
{O} {0} 10 0.92%
{n} {} 9 0.83%
{'} {} 9 0.83%
{y} {v} 9 0.83%
{.} {} 8 0.74%
The remaining but hidden errors make up 84.29%
The result has improved, but I still couldn't match the CER as indicated in the paper. Did I miss anything in my procedure?
from calamari.
18813055625
commented on June 18, 2024
@zhangxiangnick /ModernEnglish.ckpt在哪里下载？
from calamari.
ChWick
commented on June 18, 2024
@alwc Add --n_confusions -1 to calamari-eval and you should find a few lines of wrong GT (empty prediction but full string as GT) that should make up approx 25% of the remaining discrepancy.
from calamari.
alwc
commented on June 18, 2024
@ChWick So the calamari number 0.155% in Table 2 is obtained by "0.23 * (1 - sum of wrong GT %) = 0.155"?
BTW, this prediction is really weird:
GT PRED COUNT PERCENT
{49} {Foam Rubber Modeling of El Centro Terminal Substation Building} 1 5.73%
@18813055625 https://github.com/Calamari-OCR/calamari_models
from calamari.
ChWick
commented on June 18, 2024
I manually detected those lines and rotated them (all 3? lines are rotated by 90 degrees). To find them you can use the --xlsx_output flag of the eval script. Sort for highest relative error and rotate the corresponding lines.
from calamari.
commented on June 18, 2024
@zhangxiangnick According to the sample example data.png that you provided, your detected text is way off thats because the ModernEnglish.ckpt model is not trained on similar data to your example.
Instead, it can recognize modern english text without augmentation, example:
If you want Calamari to recognize images similar to the example you provided, you should train a new model to do so, by providing training data/ images that are similar to what you want to recognize.
Once you train you own recognition mode, Calamari-ocr will be able to recognize images similar to your own with high recognition rate.
Remember, this is not 1 size fits all situation.
from calamari.
commented on June 18, 2024
@alwc It's clear that you don't know what your talking about.
The images that you are testing on, are just a small sample of 50 lines from the UW3 dataset.
@ChWick Please close this Issue.
from calamari.
Related Issues (20)
- calamari-train: warmstart not working without also giving network spec
- featreq: when warmstart-training, init weights of new chars from existing ones HOT 2
- calamari-eval: skip missing pairs HOT 3
- calamari-eval: unknown arguments HOT 6
- calamari-eval: confusion table miscalculates relative frequency HOT 3
- Error when convert old trained model to latest version model HOT 1
- Got exception during training HOT 4
- calamari-ocr 2.2.2 on ubuntu 22.04 partial success, difficulty with GPU software
- Prediction from calamari trained .pb model HOT 5
- Issue while using the model and json HOT 8
- setup.py on Ubuntu20.04: tensorflow is wrong version HOT 7
- Model very sensitive on PNG input HOT 3
- calamari/1.0: hold Tensorflow and Protobuf dependencies HOT 6
- What is the accuracy on Chinese/Japanese text? HOT 2
- Attention layer
- "No training configuration" for code that should not have one HOT 5
- Downgrading of models is not supported (5 to 2). Please upgrade your Calamari instance (currently installed: 1.0.6) HOT 4
- UnknownArgumentError HOT 7
- Release confusion HOT 4
- calmari/1.0: Fix 1.0.x models for Python 3.11 HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from calamari.