implement CRNN in Keras with Spatial Transformer Network (STN) for Optical Character Recognition(OCR)

The model is easy to start a trainning, but the performance of recognition is not better than the original CRNN without STN.

You can run CRNN individually by just remove the STN components, and connect batchnorm_7 to x_shape. The CRNN can reach 90% of recognition accuracy.

Train on Synthetic Word Dataset realsed by M. Jaderberg et al. You can download the dataset HERE

Just run the CRNN_with_STN.py script. My environment is Tensorflow 1.4.0, Keras 2.0.9. If you don't know how to install frameworks, please check Installing Deep Learning Frameworks on Ubuntu with CUDA support.

Shi, Baoguang, X. Bai, and C. Yao. "An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition." IEEE Transactions on Pattern Analysis & Machine Intelligence PP.99(2016):1-1.[arxiv]

Max Jaderberg, Karen Simonyan,Andrew Zisserman and Koray Kavukcuoglu. "Spatial Transformer Network" [arxiv]

I have tried to put STN part in the front of the network, between batchnorm_1 and conv_2, than the network didn't converge at all. Maybe I used STN in a wrong way?