I have a problem that when I run the scripts in this form
python scripts/run_experiment.py trainer.gpus=4 data.batch_size=32 and it will cause a bug that is shown below

How to handle this problem?
Thanks!

I convert checkpoint export to onnx， but fail
lit_model = LitResNetTransformer.load_from_checkpoint("artifacts/model_basic_2.ckpt") # lit_model.freeze() lit_model.eval() x = torch.randn((1, 16)) lit_model.to_onnx("xxx.onnx", x)

Traceback (most recent call last): File "C:/Users/Administrator/PycharmProjects/image-to-latex/model_test.py", line 44, in <module> load_model() File "C:/Users/Administrator/PycharmProjects/image-to-latex/model_test.py", line 21, in load_model torch.onnx.export(lit_model, x, "hpocr_torch.onnx", verbose=True, input_names=input_names, File "C:\Users\Administrator\anaconda3\envs\image-to-latex\lib\site-packages\torch\onnx\__init__.py", line 275, in export return utils.export(model, args, f, export_params, verbose, training, File "C:\Users\Administrator\anaconda3\envs\image-to-latex\lib\site-packages\torch\onnx\utils.py", line 88, in export _export(model, args, f, export_params, verbose, training, input_names, output_names, File "C:\Users\Administrator\anaconda3\envs\image-to-latex\lib\site-packages\torch\onnx\utils.py", line 689, in _export _model_to_graph(model, args, verbose, input_names, File "C:\Users\Administrator\anaconda3\envs\image-to-latex\lib\site-packages\torch\onnx\utils.py", line 458, in _model_to_graph graph, params, torch_out, module = _create_jit_graph(model, args, File "C:\Users\Administrator\anaconda3\envs\image-to-latex\lib\site-packages\torch\onnx\utils.py", line 422, in _create_jit_graph graph, torch_out = _trace_and_get_graph_from_model(model, args) File "C:\Users\Administrator\anaconda3\envs\image-to-latex\lib\site-packages\torch\onnx\utils.py", line 373, in _trace_and_get_graph_from_model torch.jit._get_trace_graph(model, args, strict=False, _force_outplace=False, _return_inputs_states=True) File "C:\Users\Administrator\anaconda3\envs\image-to-latex\lib\site-packages\torch\jit\_trace.py", line 1160, in _get_trace_graph outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs) File "C:\Users\Administrator\anaconda3\envs\image-to-latex\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "C:\Users\Administrator\anaconda3\envs\image-to-latex\lib\site-packages\torch\jit\_trace.py", line 127, in forward graph, out = torch._C._create_graph_by_tracing( File "C:\Users\Administrator\anaconda3\envs\image-to-latex\lib\site-packages\torch\jit\_trace.py", line 118, in wrapper outs.append(self.inner(*trace_inputs)) File "C:\Users\Administrator\anaconda3\envs\image-to-latex\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "C:\Users\Administrator\anaconda3\envs\image-to-latex\lib\site-packages\torch\nn\modules\module.py", line 1039, in _slow_forward result = self.forward(*input, **kwargs) File "C:\Users\Administrator\anaconda3\envs\image-to-latex\lib\site-packages\pytorch_lightning\core\lightning.py", line 529, in forward return super().forward(*args, **kwargs) File "C:\Users\Administrator\anaconda3\envs\image-to-latex\lib\site-packages\torch\nn\modules\module.py", line 201, in _forward_unimplemented raise NotImplementedError

I wished for a tool like this for a long time 🤯

the \cal is been obsoleted for a while, and been replaced with \mathcal

ref:

• why-the-symbols-does-not-render-correctly

Firstly, thank you for the awesome project. I am a bit confused about why you configured the Transformer decoder like this:

  d_model: 128
  dim_feedforward: 256
  nhead: 4
  dropout: 0.3
  num_decoder_layers: 3
  max_output_len: 150

How can I configure these parameters correctly?
Thank you for reading, and I hope you will respond to me soon.
@kingyiusuen

feature request: the program has a feature with which you can get the image by screenshot instead of drag n drop

Hello and so happy to see you use Pytorch-Lightning! 🎉
Just wondering if you already heard about quite the new Pytorch Lightning (PL) ecosystem CI where we would like to invite you to... You can check out our blog post about it: Stay Ahead of Breaking Changes with the New Lightning Ecosystem CI ⚡
As you use PL framework for your cool project, we would like to enhance your experience and offer you safe updates to our future releases. At this moment, you run tests with a particular PL version, but it may accidentally happen that the next version will be incompatible with your project... 😕 We do not intend to change anything on our project side, but still here we have a solution - ecosystem CI with testing both - your and our latest development head we can find it very early and prevent releasing eventually bad version... 👍

What is needed to do?

• have some tests, including PL integration
• add config to ecosystem CI - https://github.com/PyTorchLightning/ecosystem-ci

What will you get?

• scheduled nightly testing configured for development/stable versions
• slack notification if something went wrong to investigate
• testing also on multi-GPU machine as our gift to you 🐰

cc: @Borda

If you want to obtain the coordinate position of each visible character, how can you modify the code?

Hi, your project image-to-latex requires "albumentations==1.0.3" in its dependency. After analyzing the source code, we found that the following versions of albumentations can also be suitable without affecting your project, i.e., albumentations 1.0.0, 1.0.1, 1.0.2. Therefore, we suggest to loosen the dependency on albumentations from "albumentations==1.0.3" to "albumentations>=1.0.0,<=1.0.3" to avoid any possible conflict for importing more packages or for downstream projects that may use image-to-latex.

May I pull a request to further loosen the dependency on albumentations?

By the way, could you please tell us whether such dependency analysis may be potentially helpful for maintaining dependencies easier during your development?

We also give our detailed analysis as follows for your reference:

Your project image-to-latex directly uses 5 APIs from package albumentations.

albumentations.augmentations.geometric.transforms.Affine.__init__, albumentations.pytorch.transforms.ToTensorV2.__init__, albumentations.core.composition.Compose.__init__, albumentations.augmentations.transforms.GaussianBlur.__init__, albumentations.augmentations.transforms.GaussNoise.__init__

Beginning from the 5 APIs above, 15 functions are then indirectly called, including 14 albumentations's internal APIs and 1 outsider APIs. The specific call graph is listed as follows (neglecting some repeated function occurrences).

[/kingyiusuen/image-to-latex]
+--albumentations.augmentations.geometric.transforms.Affine.__init__
|      +--albumentations.core.transforms_interface.BasicTransform.__init__
|      +--albumentations.augmentations.geometric.transforms.Affine._handle_dict_arg
|      |      +--albumentations.core.transforms_interface.to_tuple
|      +--albumentations.augmentations.geometric.transforms.Affine._handle_translate_arg
|      +--albumentations.core.transforms_interface.to_tuple
+--albumentations.pytorch.transforms.ToTensorV2.__init__
|      +--albumentations.core.transforms_interface.BasicTransform.__init__
+--albumentations.core.composition.Compose.__init__
|      +--albumentations.core.composition.BaseCompose.__init__
|      |      +--albumentations.core.composition.Transforms.__init__
|      |      |      +--albumentations.core.composition.Transforms._find_dual_start_end
|      |      |      |      +--albumentations.core.composition.Transforms._find_dual_start_end
|      +--albumentations.augmentations.bbox_utils.BboxProcessor.__init__
|      |      +--albumentations.core.utils.DataProcessor.__init__
|      +--albumentations.core.composition.BboxParams.__init__
|      |      +--albumentations.core.utils.Params.__init__
|      +--albumentations.augmentations.keypoints_utils.KeypointsProcessor.__init__
|      |      +--albumentations.core.utils.DataProcessor.__init__
|      +--albumentations.core.composition.KeypointParams.__init__
|      |      +--albumentations.core.utils.Params.__init__
|      +--albumentations.core.composition.BaseCompose.add_targets
+--albumentations.augmentations.transforms.GaussianBlur.__init__
|      +--albumentations.core.transforms_interface.BasicTransform.__init__
|      +--albumentations.core.transforms_interface.to_tuple
|      +--warnings.warn
+--albumentations.augmentations.transforms.GaussNoise.__init__
|      +--albumentations.core.transforms_interface.BasicTransform.__init__

We scan albumentations's versions and observe that during its evolution between any version from [1.0.0, 1.0.1, 1.0.2] and 1.0.3, the changing functions (diffs being listed below) have none intersection with any function or API we mentioned above (either directly or indirectly called by this project).

diff: 1.0.3(original) 1.0.0
['albumentations.core.composition.Compose._check_data_post_transform', 'albumentations.core.utils.DataProcessor.postprocess', 'albumentations.augmentations.transforms.PadIfNeeded.update_params', 'albumentations.core.composition.Compose.__call__', 'albumentations.core.composition.Compose', 'albumentations.core.utils.get_shape', 'albumentations.augmentations.transforms.Normalize', 'albumentations.augmentations.geometric.transforms.Affine', 'albumentations.augmentations.crops.transforms.CropAndPad._get_px_params', 'albumentations.augmentations.crops.transforms.CropAndPad', 'albumentations.augmentations.transforms.Superpixels', 'albumentations.augmentations.transforms.PadIfNeeded', 'albumentations.augmentations.bbox_utils.convert_bbox_from_albumentations', 'albumentations.core.utils.DataProcessor', 'albumentations.augmentations.transforms.PadIfNeeded.PositionType', 'albumentations.augmentations.transforms.PadIfNeeded.__update_position_params', 'albumentations.augmentations.bbox_utils.convert_bbox_to_albumentations', 'albumentations.augmentations.transforms.PadIfNeeded.__init__', 'albumentations.augmentations.bbox_utils.check_bbox']

diff: 1.0.3(original) 1.0.1
['albumentations.core.composition.Compose.__call__', 'albumentations.core.composition.Compose', 'albumentations.core.composition.Compose._check_data_post_transform', 'albumentations.augmentations.bbox_utils.convert_bbox_to_albumentations', 'albumentations.augmentations.transforms.Superpixels', 'albumentations.core.utils.DataProcessor.postprocess', 'albumentations.core.utils.get_shape', 'albumentations.augmentations.bbox_utils.convert_bbox_from_albumentations', 'albumentations.core.utils.DataProcessor', 'albumentations.augmentations.bbox_utils.check_bbox']

diff: 1.0.3(original) 1.0.2
['albumentations.core.composition.Compose', 'albumentations.core.composition.Compose._check_data_post_transform', 'albumentations.augmentations.transforms.Superpixels', 'albumentations.core.utils.DataProcessor.postprocess', 'albumentations.core.utils.get_shape', 'albumentations.core.utils.DataProcessor', 'albumentations.augmentations.bbox_utils.check_bbox']

As for other packages, the APIs of warnings are called by albumentations in the call graph and the dependencies on these packages also stay the same in our suggested versions, thus avoiding any outside conflict.

Therefore, we believe that it is quite safe to loose your dependency on albumentations from "albumentations==1.0.3" to "albumentations>=1.0.0,<=1.0.3". This will improve the applicability of image-to-latex and reduce the possibility of any further dependency conflict with other projects.

I downloaded the author’s best run from wandb, and after experimentation, the results are as follows:

Did I download the wrong model?

Hey guy, your work is cool! If possible, can you tell me the best metrics you got when training on im2latex-100K, such as BLEU WER , ROUGE, etc.?

i want kingyiusuen/image-to-latex/1w1abmg1，
but keep Downloading model checkpoint...

Can anyone provide a download link for this model file? thanks

Am a beginner in Code. I don't know how to run this code. Anyone please help?

it seems something wrong in process Data Preprocessing, can not download dateset in url "http://lstm.seas.harvard.edu/latex/data/formula_images.tar.gz"

make : The term 'make' is not recognized as the name of a cmdlet, function, script file, or operable program. Check
the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1

• make venv

•   + CategoryInfo          : ObjectNotFound: (make:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException
  

这是怎么回事

it took so long time to run training of just one epoch, as My PC doesn't have GPU.

Dear author:
I am a freshman in github and sorry to bother you with the error I meet which may seem to be quiet wired for you. When I want to download the repositories to my local computer, with the third command you offered in the "how to use" section, both vscode and python report the errors as follow:

could you explain why and how to solve the problem?
Thanks a lot

origin

predict

If I want to get the position of each recognized character on the original image, how can I modify the model?

I got CER of 0.06 on im2latex-100K.
But I can't get good results from the API.
Please give me some advice.

ls -l checkpoints

drwxr-xr-x 3 root root 4096 Apr 13 15:40 'epoch=1-val'
drwxr-xr-x 3 root root 4096 Apr 13 15:54 'epoch=3-val'
drwxr-xr-x 3 root root 4096 Apr 13 16:07 'epoch=5-val'
drwxr-xr-x 3 root root 4096 Apr 13 16:21 'epoch=7-val'
drwxr-xr-x 3 root root 4096 Apr 13 16:34 'epoch=9-val'
drwxr-xr-x 3 root root 4096 Apr 13 16:48 'epoch=11-val'
drwxr-xr-x 3 root root 4096 Apr 13 17:01 'epoch=13-val'

ls -l epoch=13-val

drwxr-xr-x 2 root root 4096 Apr 15 09:42 'loss=0.12-val'

ls -l loss=0.12-val

-rw-r--r-- 1 root root 2062314067 Apr 13 17:01 'cer=0.06.ckpt'

vi test_predictions.txt

\alpha _ { 1 } ^ { r } \gamma _ { 1 } + \ldots + \alpha _ { N } ^ { r } \gamma _ { N } = 0 \quad ( r = 1 , . . . , R ) \ ,
\eta = - \frac { 1 } { 2 } \operatorname { l n } ( \frac { \operatorname { c o s h } ( \sqrt { 2 } b _ { \infty } \sqrt { 1 + \alpha ^ { 2 } } y - \operatorname { a r c s i n h } \alpha ) } { \sqrt { 1 + \alpha ^ { 2 } } } } } )
P _ { ( 2 ) } ^ { - } = \int \beta d \beta d ^ { 9 } p d ^ { 8 } \lambda \Phi ( - p , - \lambda ) ( - \frac { p ^ { I } p ^ { I } } { 2 \beta } ) \Phi ( p , \lambda ) \Phi ( p , \lambda ) \Phi ( p , \lambda ) \Phi ( p , \lambda ) \Phi ( p , \lambda ) \Phi ( p , \lambda ) \Phi ( p , \lambda ) \Phi ( p , \lambda ) \Phi ( p , \lambda ) \Phi ( p , \lambda ) \Phi ( p , \lambda ) \Phi ( p , \lambda ) \Phi ( p , \lambda ) \Phi ( p , \lambda ) \Phi ( p ,
\Gamma ( z + 1 ) = \int _ { 0 } ^ { \infty } ; d x ; e ^ { - x } x ^ { z } .
\frac { d } { d s } { \bf C } _ { i } = \frac { 1 } { 2 } \epsilon _ { i j k } { \bf C } _ { j } \times { \bf C } _ { k } , .

API Test

curl -X 'POST' 'http://0.0.0.0:8000/predict/' -H 'accept: application/json' -H 'Content-Type: multipart/form-data' -F '[email protected];type=image/png'

{"message":"OK","status-code":200,"data":{"pred":"E = m c ^ { 2 } \qquad \qquad E = m c ^ { 2 } \qquad E = m c ^ { 2 }"}}

curl -X 'POST' 'http://0.0.0.0:8000/predict/' -H 'accept: application/json' -H 'Content-Type: multipart/form-data' -F '[email protected];type=image/png'

{"message":"OK","status-code":200,"data":{"pred":"{ I } _ { \mathrm { I } } { \mit { U } } } ( { \bf { I } } { \bf { U } } ) } ) \; \; \Longrightarrow \; { \bf { U } } { \binom { \bf { I } } } { { \bf { H } } } ) \; { \frac { 1 } { \longrightarrow } } \; { \bf { U } } } { \bf { I } } } \Bigg \{ { \bf { I } } } { \bf { I } } ) \; { \bf { T } } \; { \bf { U } } { \bf { U } } \; { \bf { U } } { \bf { U } } } \; { \bf { U } } { \bf { U } } { \bf"}}

curl -X 'POST' 'http://0.0.0.0:8000/predict/' -H 'accept: application/json' -H 'Content-Type: multipart/form-data' -F '[email protected];type=image/png'

{"message":"OK","status-code":200,"data":{"pred":"{ \bf { I } } ~ \longrightarrow ~ { \bf { I } } _ { I } \prod _ { I } \bigoplus _ { I } \bigoplus _ { \bf { I } } \bigoplus _ { \bf { O } } { \bf { I } } { \bf { I } } } { \bf { I } } { \bf { I } } { \bf { I } } } { \bf { I } } } { \bf { I } } { \bf { I } } } { \bf { I } } { \bf { I } } { \bf { I } } } { \bf { I } } } { \bf { I } } } { \bf { I } } } } { \bf { I } } } { \bf { I } }"}}

curl -X 'POST' 'http://0.0.0.0:8000/predict/' -H 'accept: application/json' -H 'Content-Type: multipart/form-data' -F '[email protected];type=image/png'

{"message":"OK","status-code":200,"data":{"pred":"\begin{array} { c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c"}}

When I run

python scripts/prepare_data.py

finally print

Cleaning data...
sh: /Users/xxxxx/xxxxx/image-to-latex/data/scripts/find_and_replace.sh: No such file or directory

But I find out that find_and_replace.sh the file located in

/Users/xxxxx/xxxxx/image-to-latex/scripts/find_and_replace.sh

The file path is wrong.