Input data about cutie HOT 32 OPEN

@Hrishkesh , @sathvikask0, @Karthik1904
Guys, I have written a simple file to run Tesseract ocr and output a json file in the format as in invoice_data/ example:
https://github.com/hhien/tesseract_applications.git

from cutie.

Hi,

Can you please provide details on how to generate and test the model? There are a whole lot of files and command line arguments to be given. Can you please update README as to how this model has to be trained and tested?

from cutie.

@Ibmaria
Hello 4kssoft,
Please how do you get this format ?.I have no idea .Can you explain me please?
Thanks in advance

To get the format. I analyzed this file https://github.com/vsymbol/CUTIE/blob/master/data_loader_json.py

@samhita-alla

I'm training a model with these parameters:

python main_train_json.py \ --doc_path 'invoice_data/' \ --save_prefix 'INVOICE' \ --test_path '' \ --embedding_file '' \ --ckpt_path 'graph/' \ --ckpt_file 'CUTIE_highresolution_8x_d20000c9(r80c80_iter_40000.ckpt' \ --tokenize True \ --update_dict True \ --dict_path 'dict/' \ --rows_segment 72 \ --cols_segment 72 \ --augment_strategy 1 \ --positional_mapping_strategy 1 \ --rows_target 64 \ --cols_target 64 \ --rows_ulimit 80 \ --fill_bbox False \ --data_augmentation_extra True \ --data_augmentation_dropout 1 \ --data_augmentation_extra_rows 16 \ --data_augmentation_extra_cols 16 \ --batch_size 32 \ --iterations 40000 \ --lr_decay_step 13000 \ --learning_rate 0.0001 \ --lr_decay_factor 0.1 \ --hard_negative_ratio 3 \ --use_ghm 0 \ --ghm_bins 30 \ --ghm_momentum 0 \ --log_path 'log/' \ --log_disp_step 100 \ --log_save_step 100 \ --validation_step 100 \ --test_step 400 \ --ckpt_save_step 50 \ --embedding_size 128 \ --weight_decay 0.0005 \ --eps 1e-6

Could you please provide the ckpt file CUTIE_highresolution_8x_d20000c9(r80c80_iter_40000.ckpt and the invoice dataset that you had used for training?

from cutie.

Hello @vsymbol
Please how do you get this format ?.I have no idea .Can you explain me please?
Thanks in advance

from cutie.

Hello @vsymbol , http://52.193.30.103 seems to be down. Could you provide the updated link?

from cutie.

@4kssoft
Thanks a lot for sharing how to train the model . however how engine(api) did you use to get the boxes coordinates from the images ?
Thanks

I use own software for labeling documents (https://www.youtube.com/watch?v=1okRMNxC0ec)

from cutie.

from cutie.

Hello,
I couldn't find the exact input data to train the model on the icdar dataset , can you provide explanations for it ?
thanks.

from cutie.

Hello,
Could you provide your input data for the model to reproduce the results or at least the input data format so that I can try the model on my custom dataset

The project is refreshed with all history removed. All programs are runnable expect that the data example is not uploaded.

You may infer the correct data format from the data_loader_json.py file. Pull request is welcomed for making the project runnable out of the box. I'll let you know when the original data format can be provided, otherwise please feel free to create a pull request.

from cutie.

Please tell me if this is the correct data format?

Format:

file_name.json

{
  "global_attributes": {
    "file_id": "$file_name"
  },
  "fields":[
    {
      "field_name": "$class_name",
      "key_id": [],
      "key_text": [],
      "value_id": [$word_id],
      "value_text": "$word_text"
    },...
  ],

  "text_boxes":[
    {
      "id": $word_id,
      "bbox": [$word_x_min, $word_y_min, $word_x_max, $word_y_max],
      "text": "$word_text"
    },...
  ]
}

Example:

file1.json

{
  "global_attributes": {
    "file_id": "file1.jpg"
  },
  "fields":[
    {
      "field_name": "class1",
      "key_id": [],
      "key_text": [],
      "value_id": [1],
      "value_text": "sample1"
    },
    {
      "field_name": "class1",
      "key_id": [],
      "key_text": [],
      "value_id": [2],
      "value_text": "sample2"
    },
    {
      "field_name": "class2",
      "key_id": [],
      "key_text": [],
      "value_id": [3],
      "value_text": "sample3"
    }
  ],

  "text_boxes":[
    {
      "id": 1,
      "bbox": [10, 10, 50, 20],
      "text": "sample1"
    },
    {
      "id": 2,
      "bbox": [55, 10, 100, 20],
      "text": "sample2"
    },
    {
      "id": 3,
      "bbox": [50, 30, 100, 40],
      "text": "sample3"
    }
  ]
}

Or maybe the correct format should look like this
Example 2:

file1.json

{
  "global_attributes": {
    "file_id": "file1.jpg"
  },
  "fields":[
    {
      "field_name": "class1",
      "key_id": [],
      "key_text": [],
      "value_id": [1, 2],
      "value_text": ["sample1", "sample2"]
    },
    {
      "field_name": "class2",
      "key_id": [],
      "key_text": [],
      "value_id": [3],
      "value_text": ["sample3"]
    }
  ],

  "text_boxes":[
    {
      "id": 1,
      "bbox": [10, 10, 50, 20],
      "text": "sample1"
    },
    {
      "id": 2,
      "bbox": [55, 10, 100, 20],
      "text": "sample2"
    },
    {
      "id": 3,
      "bbox": [50, 30, 100, 40],
      "text": "sample3"
    }
  ]
}

from cutie.

Hello @vsymbol , http://52.193.30.103 seems to be down. Could you provide the updated link?

I too am unable to reach 52.193.30.103 even via ping. Can you confirm if this is up?

from cutie.

Sample data file #8 (comment)

from cutie.

Hello @vsymbol , http://52.193.30.103 seems to be down. Could you provide the updated link?

I too am unable to reach 52.193.30.103 even via ping. Can you confirm if this is up?

Hi varshaneya, the link is down.

from cutie.

@Ibmaria
Hello 4kssoft,
Please how do you get this format ?.I have no idea .Can you explain me please?
Thanks in advance

To get the format. I analyzed this file https://github.com/vsymbol/CUTIE/blob/master/data_loader_json.py

@samhita-alla

I'm training a model with these parameters:

python main_train_json.py \ --doc_path 'invoice_data/' \ --save_prefix 'INVOICE' \ --test_path '' \ --embedding_file '' \ --ckpt_path 'graph/' \ --ckpt_file 'CUTIE_highresolution_8x_d20000c9(r80c80_iter_40000.ckpt' \ --tokenize True \ --update_dict True \ --dict_path 'dict/' \ --rows_segment 72 \ --cols_segment 72 \ --augment_strategy 1 \ --positional_mapping_strategy 1 \ --rows_target 64 \ --cols_target 64 \ --rows_ulimit 80 \ --fill_bbox False \ --data_augmentation_extra True \ --data_augmentation_dropout 1 \ --data_augmentation_extra_rows 16 \ --data_augmentation_extra_cols 16 \ --batch_size 32 \ --iterations 40000 \ --lr_decay_step 13000 \ --learning_rate 0.0001 \ --lr_decay_factor 0.1 \ --hard_negative_ratio 3 \ --use_ghm 0 \ --ghm_bins 30 \ --ghm_momentum 0 \ --log_path 'log/' \ --log_disp_step 100 \ --log_save_step 100 \ --validation_step 100 \ --test_step 400 \ --ckpt_save_step 50 \ --embedding_size 128 \ --weight_decay 0.0005 \ --eps 1e-6

from cutie.

@4kssoft
Thanks a lot for sharing how to train the model . however how engine(api) did you use to get the boxes coordinates from the images ?
Thanks

from cutie.

@4kssoft
Thanks you !

from cutie.

@4kssoft

@4kssoft
Thanks a lot for sharing how to train the model . however how engine(api) did you use to get the boxes coordinates from the images ?
Thanks

I use own software for labeling documents (https://www.youtube.com/watch?v=1okRMNxC0ec)

@4kssoft Thanks for sharing. How can i access you tool ?

from cutie.

@4kssoft

@4kssoft
Thanks a lot for sharing how to train the model . however how engine(api) did you use to get the boxes coordinates from the images ?
Thanks

I use own software for labeling documents (https://www.youtube.com/watch?v=1okRMNxC0ec)

@4kssoft Thanks for sharing. How can i access you tool ?

This is a beta version for now. I plan to publish this software, but not as open source

from cutie.

@Ibmaria
Hello 4kssoft,
Please how do you get this format ?.I have no idea .Can you explain me please?
Thanks in advance

To get the format. I analyzed this file https://github.com/vsymbol/CUTIE/blob/master/data_loader_json.py

@samhita-alla

I'm training a model with these parameters:

python main_train_json.py \ --doc_path 'invoice_data/' \ --save_prefix 'INVOICE' \ --test_path '' \ --embedding_file '' \ --ckpt_path 'graph/' \ --ckpt_file 'CUTIE_highresolution_8x_d20000c9(r80c80_iter_40000.ckpt' \ --tokenize True \ --update_dict True \ --dict_path 'dict/' \ --rows_segment 72 \ --cols_segment 72 \ --augment_strategy 1 \ --positional_mapping_strategy 1 \ --rows_target 64 \ --cols_target 64 \ --rows_ulimit 80 \ --fill_bbox False \ --data_augmentation_extra True \ --data_augmentation_dropout 1 \ --data_augmentation_extra_rows 16 \ --data_augmentation_extra_cols 16 \ --batch_size 32 \ --iterations 40000 \ --lr_decay_step 13000 \ --learning_rate 0.0001 \ --lr_decay_factor 0.1 \ --hard_negative_ratio 3 \ --use_ghm 0 \ --ghm_bins 30 \ --ghm_momentum 0 \ --log_path 'log/' \ --log_disp_step 100 \ --log_save_step 100 \ --validation_step 100 \ --test_step 400 \ --ckpt_save_step 50 \ --embedding_size 128 \ --weight_decay 0.0005 \ --eps 1e-6

@4kssoft if possible please provide the pretrained model that you are using!

And guys for the annotation with bounding boxes please look into this link, might be useful :
Tesseract OCR: Text localization and detection

from cutie.

@4kssoft Hi I have my own data and extracted text using OCR tesseract and got the position of each word, can i know how to get in the format you showed an example in your repository for sample pdf file Faktura1.pdf_0.json how to get in this format and i need in the format you done can you let me know ???

from cutie.

@4kssoft Thanks for your suggestions, I have generated my own training datasets and i am able to train the model, but I am not getting what should be the input format to predict the result. If you know what modification it requires to get the result please just inform us.

from cutie.

Hello all

@4kssoft if possible please provide the pretrained model that you are using!

@sathvikask0
Sorry but unfortunately I cannot share my model

@4kssoft Hi I have my own data and extracted text using OCR tesseract and got the position of each word, can i know how to > get in the format you showed an example in your repository for sample pdf file Faktura1.pdf_0.json how to get in this format > > and i need in the format you done can you let me know ???

@Hrishkesh
as I wrote #7 (comment) earlier, I use my own tool to annotate documents, I also have ready functions for exporting training data to various models.
I'm planning to publish a beta version of my solution soon

@4kssoft Thanks for your suggestions, I have generated my own training datasets and i am able to train the model, but I am >not getting what should be the input format to predict the result. If you know what modification it requires to get the result ?>please just inform us.

@Neelesh1121
The format is the same as for training. Look at the https://github.com/vsymbol/CUTIE/blob/master/main_evaluate_json.py script

from cutie.

everytime i try to use main_evaluate_json.py I get this error

@4kssoft @samhita-alla @vsymbol
Can anyone please help

2 root error(s) found.
(0) Not found: Key feature_fuser/biases not found in checkpoint
[[node save/RestoreV2 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
(1) Not found: Key feature_fuser/biases not found in checkpoint
[[node save/RestoreV2 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
[[save/RestoreV2/_49]]
0 successful operations.
0 derived errors ignored.

from cutie.

@Ibmaria
Hello 4kssoft,
Please how do you get this format ?.I have no idea .Can you explain me please?
Thanks in advance

To get the format. I analyzed this file https://github.com/vsymbol/CUTIE/blob/master/data_loader_json.py

@samhita-alla

I'm training a model with these parameters:

python main_train_json.py \ --doc_path 'invoice_data/' \ --save_prefix 'INVOICE' \ --test_path '' \ --embedding_file '' \ --ckpt_path 'graph/' \ --ckpt_file 'CUTIE_highresolution_8x_d20000c9(r80c80_iter_40000.ckpt' \ --tokenize True \ --update_dict True \ --dict_path 'dict/' \ --rows_segment 72 \ --cols_segment 72 \ --augment_strategy 1 \ --positional_mapping_strategy 1 \ --rows_target 64 \ --cols_target 64 \ --rows_ulimit 80 \ --fill_bbox False \ --data_augmentation_extra True \ --data_augmentation_dropout 1 \ --data_augmentation_extra_rows 16 \ --data_augmentation_extra_cols 16 \ --batch_size 32 \ --iterations 40000 \ --lr_decay_step 13000 \ --learning_rate 0.0001 \ --lr_decay_factor 0.1 \ --hard_negative_ratio 3 \ --use_ghm 0 \ --ghm_bins 30 \ --ghm_momentum 0 \ --log_path 'log/' \ --log_disp_step 100 \ --log_save_step 100 \ --validation_step 100 \ --test_step 400 \ --ckpt_save_step 50 \ --embedding_size 128 \ --weight_decay 0.0005 \ --eps 1e-6

@4kssoft Do you generate your own dictionnary? I don't really understand the part "Generate your own dictionary with main_build_dict.py / main_data_tokenizer.py". Can you explain how to apply this process on own dataset? Thanks

Also, to what the ckpt_path argument refers to?

from cutie.

Hello @vsymbol

can you please give brief about how to generate the texts and corresponding bounding boxes & manually labelling each text and their bounding box

Which tools we have use for manually labelling

from cutie.

@4kssoft Thanks for the labeling video. Does your software export in the format required by CUTIE (json template you provided) or you have to run explicit post processing ?
In the json example you provided, what does "key_id" and "key_value" represent ? all of them look empty.

from cutie.

Hello @vsymbol

can you please give brief about how to generate the texts and corresponding bounding boxes & manually labelling each text and their bounding box

Which tools we have use for manually labelling

Apply any OCR tool that help you detecting and recognizing words in the scanned document image.
For example, refer to what @4kssoft has done to the document image and generated a .json file with position and text of the image.

https://github.com/4kssoft/CUTIE/blob/master/invoice_data/Faktura1.pdf_0.json

from cutie.

Did someone able to train and test the model? I couldn't find how to predict on new data .

from cutie.

Did someone able to train and test the model? I couldn't find how to predict on new data .

I'm struggling with it. So far I was able to create the .json-files, with the solution of hhien's code.

I anyone succeeded, I'm thankful for any recommendation on how to train the model.

from cutie.

Please tell me if this is the correct data format?

Format:

file_name.json

{
  "global_attributes": {
    "file_id": "$file_name"
  },
  "fields":[
    {
      "field_name": "$class_name",
      "key_id": [],
      "key_text": [],
      "value_id": [$word_id],
      "value_text": "$word_text"
    },...
  ],

  "text_boxes":[
    {
      "id": $word_id,
      "bbox": [$word_x_min, $word_y_min, $word_x_max, $word_y_max],
      "text": "$word_text"
    },...
  ]
}

Example:

file1.json

{
  "global_attributes": {
    "file_id": "file1.jpg"
  },
  "fields":[
    {
      "field_name": "class1",
      "key_id": [],
      "key_text": [],
      "value_id": [1],
      "value_text": "sample1"
    },
    {
      "field_name": "class1",
      "key_id": [],
      "key_text": [],
      "value_id": [2],
      "value_text": "sample2"
    },
    {
      "field_name": "class2",
      "key_id": [],
      "key_text": [],
      "value_id": [3],
      "value_text": "sample3"
    }
  ],

  "text_boxes":[
    {
      "id": 1,
      "bbox": [10, 10, 50, 20],
      "text": "sample1"
    },
    {
      "id": 2,
      "bbox": [55, 10, 100, 20],
      "text": "sample2"
    },
    {
      "id": 3,
      "bbox": [50, 30, 100, 40],
      "text": "sample3"
    }
  ]
}

Or maybe the correct format should look like this
Example 2:

file1.json

{
  "global_attributes": {
    "file_id": "file1.jpg"
  },
  "fields":[
    {
      "field_name": "class1",
      "key_id": [],
      "key_text": [],
      "value_id": [1, 2],
      "value_text": ["sample1", "sample2"]
    },
    {
      "field_name": "class2",
      "key_id": [],
      "key_text": [],
      "value_id": [3],
      "value_text": ["sample3"]
    }
  ],

  "text_boxes":[
    {
      "id": 1,
      "bbox": [10, 10, 50, 20],
      "text": "sample1"
    },
    {
      "id": 2,
      "bbox": [55, 10, 100, 20],
      "text": "sample2"
    },
    {
      "id": 3,
      "bbox": [50, 30, 100, 40],
      "text": "sample3"
    }
  ]
}

what are bbox entries? x1,y1,widht,height? or x1,y1(top left),x2,y2(bottom right)

from cutie.

i have created the json files in the required format. i have 400 invoices data. the main_train_json.py gets killed because it utilises all the RAM. has anyone faced this issue? I have 16 gb of ram.

from cutie.

Anyone pls share the inference script?

from cutie.