irfanicmll / auto_painter Goto Github PK

Recently, realistic image generation using deep neural networks has become a hot topic in machine learning and computer vision. Such an image can be generated at pixel level by learning from a large collection of images. Learning to generate colorful cartoon images from black-and-white sketches is not only an interesting research problem, but also a useful application in digital entertainment. In this paper, we investigate the sketch-to-image synthesis problem by using conditional generative adversarial networks (cGAN). We propose a model called auto-painter which can automatically generate compatible colors given a sketch. Wasserstein distance is used in training cGAN to overcome model collapse and enable the model converged much better. The new model is not only capable of painting hand-draw sketch with compatible colors, but also allowing users to indicate preferred colors. Experimental results on different sketch datasets show that the auto-painter performs better than other existing image-to-image methods.

Python 100.00%

auto_painter's Introduction

Auto_painter

News

We have released our dataset for public use. The dataset can be downloaded through following links:

Sketch-image pairs: https://cloudstor.aarnet.edu.au/plus/s/rMSBYCjEZJ70ab2

Sketch with control color blocks: https://cloudstor.aarnet.edu.au/plus/s/ixj8XS0rMmUqq0Z

Orginal README

It is the original implementation of the journal article: Auto-painter: Cartoon image generation from sketch by using conditional Wasserstein generative adversarial networks https://www.sciencedirect.com/science/article/pii/S0925231218306209?via%3Dihub

This project mean to make an end-to-end network for the sketch of cartoon to have color automatically.

Try our demo here: http://103.202.133.77:10086/

Since the lab's server has temporarily expired, the demo is now unavailable. You can see the demo video and train your own model. Or you can build your demo page based on our provided models following this project: https://github.com/irfanICMLL/Auto_painter_demo

New model has been updated!~ The performance is much better than in the orginal paper! See the demo video:https://youtu.be/g9rf-YFGgbg

Have a try~

The pre-trained model can be downloaded from the following link: https://cloudstor.aarnet.edu.au/plus/s/LvyREKsiaH47Aa6

My homepage: https://irfanicmll.github.io/

Welcome to contact me~

Dependencies

python3.5

tensorflow1.4

Vgg model from:https://github.com/machrisaa/tensorflow-vgg(optional, if you use the loss_f)

Data

Color images: Collected on the Internet

Sketch: Generated from the preprocessing/gen_sketch/sketch.py

Quick start

Put you orginal data in the folder preprocessing/gen_sketch/pic_org

Run the sketch.py and you will get the training set in the preprocessing/gen_sketch/pic_sketch folder

Download the pre-train weight of Vgg16, and put the model and the pretrian weight uder the folder of training&test/my_vgg

Run the training command as:

python auto-painter.py --mode train --input_dir $TRAINING_SET --output_dir $OUTPUT --checkpoint None

Run the testing command as:

python auto-painter.py --mode test --input_dir $TESTING_SET --output_dir $OUTPUT_TEST --checkpoint $OUTPUT

auto_painter's People

Contributors

Stargazers

Watchers

auto_painter's Issues

Resize the output image to original size.

Do you have code for resizing the size of output images that are 512 by 512 to the size of original images?

The pre-trained model can't be loaded

When I tried loading the pre-trained model, something went wrong like below. It seems that the model's parameters doesn't match the structure?

Can anyone help? Thank you very much!

2020-01-04 20:49:38.604178: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_7/batchnorm/offset/Adam_1 not found in checkpoint
2020-01-04 20:49:38.604178: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_7/batchnorm/scale/Adam not found in checkpoint
2020-01-04 20:49:38.604221: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_7/conv/filter/Adam not found in checkpoint
2020-01-04 20:49:38.604253: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_7/batchnorm/scale/Adam_1 not found in checkpoint
2020-01-04 20:49:38.604195: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_7/batchnorm/offset/Adam not found in checkpoint
2020-01-04 20:49:38.606036: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_8/batchnorm/offset/Adam_1 not found in checkpoint
2020-01-04 20:49:38.606443: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_7/conv/filter/Adam_1 not found in checkpoint
2020-01-04 20:49:38.606492: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_8/batchnorm/offset/Adam not found in checkpoint
2020-01-04 20:49:38.607934: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_8/batchnorm/scale/Adam not found in checkpoint
2020-01-04 20:49:38.612170: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_7/batchnorm/offset/Adam not found in checkpoint
[[Node: save/RestoreV2_151 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2_151/tensor_names, save/RestoreV2_151/shape_and_slices)]]
2020-01-04 20:49:38.612178: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_7/batchnorm/offset/Adam not found in checkpoint
[[Node: save/RestoreV2_151 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2_151/tensor_names, save/RestoreV2_151/shape_and_slices)]]
2020-01-04 20:49:38.627436: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_7/batchnorm/offset/Adam not found in checkpoint
[[Node: save/RestoreV2_151 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2_151/tensor_names, save/RestoreV2_151/shape_and_slices)]]
2020-01-04 20:49:38.627436: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_7/batchnorm/offset/Adam not found in checkpoint
[[Node: save/RestoreV2_151 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2_151/tensor_names, save/RestoreV2_151/shape_and_slices)]]

how to use the pretrained model to color the sketch of the cartoon

could give me a example ,thx

why pictures tested out are all green?

i have tried two ways in training, one is training directly on the datasets, another is first using pre-trained model offered by the author then training on the datasets. both of them are tested out all green outputs as follows, one is input picture, another is output. can anybody tell me why. thanks a lot.

I downloaded VGG16 model from https://github.com/machrisaa/tensorflow-vgg and put it in my_vgg folder. However, when I run the program, it shows the log like this. Could you help me with this error? Thank you.

/usr/bin/python2.7 /data2/texttophoto/Auto_painter/training&test/auto-painter.py --mode train --input_dir /data2/texttophoto/Auto_painter/preprocessing/gen_sketch/pic_sketch/ --output_dir /data2/texttophoto/Auto_painter/preprocessing/gen_sketch/pic_color/ --checkpoint None
aspect_ratio = 1.0
batch_size = 1
beta1 = 0.5
checkpoint = None
display_freq = 0
f_weight = 0.0001
flip = True
gan_weight = 1
input_dir = /data2/texttophoto/Auto_painter/preprocessing/gen_sketch/pic_sketch/
l1_weight = 50.0
lr = 2e-05
max_epochs = 20
max_steps = None
mode = train
ndf = 48
ngf = 48
output_dir = /data2/texttophoto/Auto_painter/preprocessing/gen_sketch/pic_color/
output_filetype = png
progress_freq = 50
save_freq = 5000
scale_size = 530
seed = 1242561248
summary_freq = 100
trace_freq = 0
tv_weight = 0.001
examples count = 819
/data2/texttophoto/Auto_painter/training&test/my_vgg/vgg16.npy
npy file loaded
/data2/texttophoto/Auto_painter/training&test/my_vgg/vgg16.npy
npy file loaded
Traceback (most recent call last):
File "/data2/texttophoto/Auto_painter/training&test/auto-painter.py", line 714, in
main()
File "/data2/texttophoto/Auto_painter/training&test/auto-painter.py", line 549, in main
model = create_model(examples.inputs, examples.targets, net1, net2)
File "/data2/texttophoto/Auto_painter/training&test/auto-painter.py", line 373, in create_model
gen_loss_f = tf.reduce_mean(tf.abs(feature_loss(targets,net1) - feature_loss(outputs,net2)))
File "/data2/texttophoto/Auto_painter/training&test/auto-painter.py", line 335, in feature_loss
vgg.build(image)
File "/data2/texttophoto/Auto_painter/training&test/my_vgg/vgg16.py", line 36, in build
assert red.get_shape().as_list()[1:] == [224, 224, 1]
AssertionError
build model started
Process finished with exit code 1