I've dug into this problem now a few times and it appears to be non-trivial and a product of drawing each letter separately with pygame. I hope there's a way around this, however.

What was the number of bounding boxes that you were predicting per grid cell ?

Thanks!

This code can not run in opencv 3.1. There have any version of this code which can run in opencv 3.1?

How to get the "seg" "label" and "area" from gPb-UCM (https://github.com/jponttuset/mcg) code that you advised. I only found it got ucm image. Are these information including "seg" "label" and "area" related to superpixel? Can you give me a clue about the specific process gathering these information? Thanks a lot. @ankush-me

how can i use this to generate cineses word image

i meet this problem,
xting@xiaomi-To-be-filled-by-O-E-M:~/SynthText$ python gen.py
getting data..
-> done
Storing the output in: results/SynthText.h5
0 of 4
Traceback (most recent call last):
File "/home/xting/SynthText/synthgen.py", line 615, in render_text
regions = self.filter_for_placement(xyz,seg,regions)
File "/home/xting/SynthText/synthgen.py", line 391, in filter_for_placement
res = get_text_placement_mask(xyz,seg==l,regions['coeff'][idx],pad=2)
File "/home/xting/SynthText/synthgen.py", line 219, in get_text_placement_mask
contour,hier = cv2.findContours(mask.copy().astype('uint8'),mode=cv2.RETR_CCOMP,method=cv2.CHAIN_APPROX_SIMPLE)
ValueError: too many values to unpack
1 of 4
ALSA lib pcm.c:7843:(snd_pcm_recover) underrun occurred
Traceback (most recent call last):
File "/home/xting/SynthText/synthgen.py", line 615, in render_text
regions = self.filter_for_placement(xyz,seg,regions)
File "/home/xting/SynthText/synthgen.py", line 391, in filter_for_placement
res = get_text_placement_mask(xyz,seg==l,regions['coeff'][idx],pad=2)
File "/home/xting/SynthText/synthgen.py", line 219, in get_text_placement_mask
contour,hier = cv2.findContours(mask.copy().astype('uint8'),mode=cv2.RETR_CCOMP,method=cv2.CHAIN_APPROX_SIMPLE)
ValueError: too many values to unpack
2 of 4
Traceback (most recent call last):
File "/home/xting/SynthText/synthgen.py", line 615, in render_text
regions = self.filter_for_placement(xyz,seg,regions)
File "/home/xting/SynthText/synthgen.py", line 391, in filter_for_placement
res = get_text_placement_mask(xyz,seg==l,regions['coeff'][idx],pad=2)
File "/home/xting/SynthText/synthgen.py", line 219, in get_text_placement_mask
contour,hier = cv2.findContours(mask.copy().astype('uint8'),mode=cv2.RETR_CCOMP,method=cv2.CHAIN_APPROX_SIMPLE)
ValueError: too many values to unpack
3 of 4
ALSA lib pcm.c:7843:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7843:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7843:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7843:(snd_pcm_recover) underrun occurred
Traceback (most recent call last):
File "/home/xting/SynthText/synthgen.py", line 615, in render_text
regions = self.filter_for_placement(xyz,seg,regions)
File "/home/xting/SynthText/synthgen.py", line 391, in filter_for_placement
res = get_text_placement_mask(xyz,seg==l,regions['coeff'][idx],pad=2)
File "/home/xting/SynthText/synthgen.py", line 219, in get_text_placement_mask
contour,hier = cv2.findContours(mask.copy().astype('uint8'),mode=cv2.RETR_CCOMP,method=cv2.CHAIN_APPROX_SIMPLE)
ValueError: too many values to unpack
4 of 4
Traceback (most recent call last):
File "/home/xting/SynthText/synthgen.py", line 615, in render_text
regions = self.filter_for_placement(xyz,seg,regions)
File "/home/xting/SynthText/synthgen.py", line 391, in filter_for_placement
res = get_text_placement_mask(xyz,seg==l,regions['coeff'][idx],pad=2)
File "/home/xting/SynthText/synthgen.py", line 219, in get_text_placement_mask
contour,hier = cv2.findContours(mask.copy().astype('uint8'),mode=cv2.RETR_CCOMP,method=cv2.CHAIN_APPROX_SIMPLE)
ValueError: too many values to unpack

I got error as follows when I run invert_font_size.py

File "invert_font_size.py", line 7, in
from synth2 import FontState

I'm training my network now and from epoch to epoch, I'm testing the model to see how well it is in predictions. So far, it's getting me less than 25% recall and I'm trying to figure out what I'm doing wrong. I believe the math I'm doing for converting the pose parameters back to bounding boxes is correct. Anyways, some light research brought me to stochastic pooling and I'm curious if you have ever tried this in your networks and if it's worth me trying to see if maybe my network is overfitting.

Also- how many epochs did you end up running? My epochs are each taking 30 hours with roughly 550k images (I will try a larger set once I am able to see if this approach is going to generalize well, if at all, to my actual data).

In synthgen.py, I know that pts is points of contour in 2d image and pts_fp is where these points should be in fronto-parallel. But what does pts_tmp mean?
rect = cv2.minAreaRect(pts_fp[0].copy().astype('float32'))
box = np.array(cv2.cv.BoxPoints(rect))
R2d = su.unrotate2d(box.copy())
box = np.vstack([box,box[0,:]]) #close the box for visualization

mu = np.median(pts_fp[0],axis=0)
pts_tmp = (pts_fp[0]-mu[None,:]).dot(R2d.T) + mu[None,:]
boxR = (box-mu[None,:]).dot(R2d.T) + mu[None,:]
s = rescale_frontoparallel(pts_tmp,boxR,pts[0])

Why does these codes necessary ? Why pts_fp and box cant't use in function “rescale_frontoparallel " directlly but use pts_tmp and boxR?

The size of images in SynthText are the same? And How big?

some chars is not rightly placed(maybe the reason of font), but i cannot find how to filter this in pygame

@ankush-me This is a great work. Thanks for sharing it.
Currently I am working on a problem of text recognition from cropped word images. Initially I used MJSynth dataset provided by Visual Geometry Group for training my LSTM based model. But this dataset is very biased towards words and has very less occurrences of numbers. I have a dictionary containing the type of text/numbers/symbols that I want to recognize but I am more concerned about rendering process. Can I use your script to synthetically generate cropped images (resembling natural text images) containing words and numbers?
Please help.

Hi,
Is there any trick or constraint to force the confidence in the 7 length vector to lie between 0 and 1 during training.

THanks

can push FCRN algorithm to git ?

Hi,
For the paper "Synthetic Data for Text Localisation in Natural Images" did you calculate IoU for oriented BBs or you converted them to axis aligned bbs ?

Thanks
M

It's strange and it happens quite often, maybe 20% of the time. It usually occurs when a word is rotated itself or has been projected onto a surface causing it to have a strange dimension. Thing is, though, it does not only happen on extreme projections. It also happens on only slightly projected text. It makes it hard for the fully convolutional network to converge well on the sin/cos pose params.

Hi, is there any background images database available? Crawling and manual inspecting large number of images is really annoying... Thx!

Hello ankush, I am having trouble creating my own data set. I want add images to dest.h5,but I didn't find the method. Can you provide the example?

Hi Ankur,
In your paper you mention hough voting but I do not see it being used in the paper. Can you clarify ?

Thanks

In a few cases, it almost appears the bottom left and the top right boxes are flipped causing the bounding box to almost appear as two triangles that are touching at one of the points rather than a single rectangle. Not exactly sure what is causing this

Thanks for sharing the code. I want to get some bigger text (for example, the text occupies a large part of the image), can you tell me how to operate your code? Thank you very much.

Hi,
Sorry to ask a question regarding the SynthText dataset since I am not aware of any other place to pose this q .

I am guessing that the size of the textboxes and angles are wrt the original image size and not wrt 512 x 512 ?

Thanks

I've recently been kicking around the idea of possibly bypassing the localization stage altogether and moving right into a holistic method for text recognition- more specifically for creating a saliency map of characters found within in an image. My thoughts are that instead of trying to predict the pose params for a bounding box around the text, I'm trying to predict a vector that gives the confidence that a specific cell contains each character of the alphabet.

I don't recall seeing any research papers published on this topic yet. Ankush, have you seen any research done on this? Have you tried this yourself?

Recently, I trained up a text detection network that, similar to your FCRN, only tried to detect the presence of text within each cell. Basically a 0 meant no text while a 1 meant high confidence of text. The results I've had both on the synth text dataset and on other datasets have been very promising.

I'm having a lot of fun with this FCRN approach, if you couldn't tell. For my specific case, I don't care as much about having bounding boxes around text as I do for knowing what the text in the image says.

Thanks again for any advice, comments, lessons learned, or references you may have.

First thanks for sharing the code, I want to remove some restrictions in your filtering (which use depth and segmentation masks), but so many numeric operations (like ransac, depth camera..) maybe not that intuitive to understand, can you show me a direction ? Maybe some detailed materials about how to use the depth and segmentation info will help a lot.

I found the text angle is related to plane normal, and it range from -90 to 90 degrees. Is there any possible for text angle range from -180 to 180 degrees?

This program can run correctly, but in order to get more SynthText samples, how can I get a lot of images with depth and segmentation information? @ankush-me

Hi,
I had a question about the loss function in your synthtext paper. Can you give a latex formula for the version used in synth text paper?

when i use the python gen.py --viz in the command

it generates
~/SynthText-master$ python gen.py --viz
Traceback (most recent call last):
File "gen.py", line 19, in
from synthgen import *
File "/home/ubuntu/SynthText-master/synthgen.py", line 20, in
import text_utils as tu
File "/home/ubuntu/SynthText-master/text_utils.py", line 12, in
from pygame import freetype
ImportError: cannot import name freetype

waht is the wrong ?

I have some images that have quite a bit of space open for text to be drawn but only a single 2-3 character word will be drawn on the entire image. It would be nice to specify if the minimum amount of words that I'd like drawn on an image so that it can continue looking for words to draw.

（this is not issue) is there a way to train custom data with synthtext?If yes ,is there a instructions?
thank you very much !

1. Hello, I am having trouble creating my own data set. The link to example data seems to be no longer working...
   DATA_URL = 'http://www.robots.ox.ac.uk/~ankush/data.tar.gz'

2. Question related to the textspotter demo http://zeus.robots.ox.ac.uk/textspot/
   Is this demo using your "FCRNall + multi-filt" model ? Is the code for this model available ?

Thank you

hello Ankush , I find the way

I came here after reading "Synthetic Data and Artificial Neural Networks for
Natural Scene Text Recognition" looking for the code to generate Synthetic Word Dataset (http://www.robots.ox.ac.uk/~vgg/data/text/). Is the code available?

Thanks.

Hi, I want to generate natural Chinese character images, so could you tell me that how to generate char_freq.cp and font_px2pt.cp corresponding Chinese in detail.
Thank you.

I add chinese font into data/font/ and update the fontlist.txt, then according to you I should run invert_font_size.py to update the font_px2pt.cp. But when running invert_font_size.py I have difficulty when installing synth2 in 'from synth2 import Fontstate'...what is its real name if i use conda install synth2?

Similar to your research, I'm finding that 3% of the cells in my input images contained bounding boxes and I'm starting my discount at 0.01 of the loss for the negative cells in the confidence matrix, c, during training. I am having trouble finding a good step value (for increasing the discount value), however, during training. How long did it take you to train your final network with 800,000 images? How often did you increase the discount that you applied to the loss of the negative c classes during training? What did you change it by?

i wonder if i can use python + opencv to get ucm.mat file for a given image...

In the paper "Synthetic Data for Text Localisation in Natural Images", you mention that you downscale the images by 1/2, 1/4 and 1/8 in order to pick up larger words. I'm wondering if you are able to provide more details about this. For instance, when you downscale the images, did you feed them into the same (1x512x512) network? Did you fill the empty area with black/white pixels?

This is causing some problems when determining the midpoints for the bounding boxes for the fully convolutional network. It's basically putting the midpoint in a different spot then it would have been had the bounding box tightly enclosed the text. Any ideas?

I came here after reading the CVPR 2016 paper. Is the post-processing code/models from Jaderberg et al (used in the paper) available at all ? Thanks!

hello, when i run the gen.py, i failed. i can not find the "http://www.robots.ox.ac.uk/~ankush/data.tar.gz". is there other's way to get the data? thanks.

the error is as follows:
File "gen.py", line 55
print colorize(Color.RED,'Data not found and have problems downloading.',bold=True)
^
SyntaxError: invalid syntax

by the way,800000 synthetic scene-text images,the download can not complete, the URL i tried is http://www.robots.ox.ac.uk/~vgg/data/scenetext/, is it right?
thanks a lot!

def get_nline_nchar(self,mask_size,font_height,font_width):
"""
Returns the maximum number of lines and characters which can fit
in the MASK_SIZED image.
"""
H,W = mask_size
nline = int(np.ceil(H/(2*font_height)))
nchar = int(np.floor(W/font_width))
return nline,nchar
In this function, font_height and font_width are type of pt, but H and W are type of px. Why can px be divide by pt？ what‘s difference between px and pt？

If I use my own images, how to get the depth and segmentation information for my image? Is there available code for it ?

the words in data/newsgroup/newsgroup.txt is not support utf-8, only for acsii.