Code for paper Generating Images from Captions with Attention by Elman Mansimov, Emilio Parisotto, Jimmy Ba and Ruslan Salakhutdinov; ICLR 2016.
We introduce a model that generates image blobs from natural language descriptions. The proposed model iteratively draws patches on a canvas, while attending to the relevant words in the description.
The code is written in python. To use it you will need:
- Python 2.7
- Theano 0.7 (mostly tested using commit from June/July 2015)
- numpy and scipy
- h5py (HDF5 (>= 1.8.11))
- skip-thoughts
Before running the code make sure that you set floatX to float32 in Theano settings.
Additionally, depending on the tasks you will probably need to download these files by running:
wget http://www.cs.toronto.edu/~emansim/datasets/mnist.h5
wget http://www.cs.toronto.edu/~emansim/datasets/text2image/train-images-32x32.npy
wget http://www.cs.toronto.edu/~emansim/datasets/text2image/train-images-56x56.npy
wget http://www.cs.toronto.edu/~emansim/datasets/text2image/train-captions.npy
wget http://www.cs.toronto.edu/~emansim/datasets/text2image/train-captions-len.npy
wget http://www.cs.toronto.edu/~emansim/datasets/text2image/train-cap2im.pkl
wget http://www.cs.toronto.edu/~emansim/datasets/text2image/dev-images-32x32.npy
wget http://www.cs.toronto.edu/~emansim/datasets/text2image/dev-images-56x56.npy
wget http://www.cs.toronto.edu/~emansim/datasets/text2image/dev-captions.npy
wget http://www.cs.toronto.edu/~emansim/datasets/text2image/dev-captions-len.npy
wget http://www.cs.toronto.edu/~emansim/datasets/text2image/dev-cap2im.pkl
wget http://www.cs.toronto.edu/~emansim/datasets/text2image/test-images-32x32.npy
wget http://www.cs.toronto.edu/~emansim/datasets/text2image/test-captions.npy
wget http://www.cs.toronto.edu/~emansim/datasets/text2image/test-captions-len.npy
wget http://www.cs.toronto.edu/~emansim/datasets/text2image/test-cap2im.pkl
wget http://www.cs.toronto.edu/~emansim/datasets/text2image/gan.hdf5
wget http://www.cs.toronto.edu/~emansim/datasets/text2image/dictionary.pkl
To train the model simply go to mnist-captions folder and run
python alignDraw.py models/mnist-captions.json
To generate 60x60 MNIST images from captions as specified in appendix of the paper run
python sample-captions.py --model models/mnist-captions.json --weights /path/to/trained-weights
Note: I have also provided implementation of simple draw model in files draw.py and sample.py
To train the model simply go to coco folder and run
python alignDraw.py models/coco-captions-32x32.json
To generate images from captions after training run
python sample-captions.py --model models/coco-captions-32x32.json --weights /path/to/trained-weights --dictionary dictionary.pkl --gan_path gan.hdf5 --skipthought_path /path/to/skipthoughts-folder
Note: I have been caught up with other non-research stuff, so I will add baseline model files like noAlignDraw and others during the week of Feb 29 - Mar 6.
Feel free to email me if you have some questions or if you are uncertain about some parts of the code.
I would like to acknowledge the help of Tom White for some suggestion on cleaning and organizing the code.
If you found this code or our paper useful, please consider citing the following paper:
@inproceedings{mansimov16_text2image,
author = {Elman Mansimov and Emilio Parisotto and Jimmy Ba and Ruslan Salakhutdinov},
title = {Generating Images from Captions with Attention},
booktitle = {ICLR},
year = {2016}
}
You would probably also need to cite some of the papers that we have referred to ;)
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent