Giter Club home page Giter Club logo

stylecarigan's Introduction

StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation

License CC BY_NC

teaser

πŸ“ This repository contains the official PyTorch implementation of the following paper:

StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation
Wonjong Jang, Gwangjin Ju, Yucheol Jung, Jiaolong Yang, Xin Tong, Seungyong Lee, SIGGRAPH 2021

πŸš€ >> Project page
πŸš€ >> Fast-forward video

Overview

method

Explanation
The key component of our method is shape exaggeration blocks that are used for modulating coarse layer feature maps of StyleGAN to produce desirable caricature shape exaggerations. We first build a layer-mixed StyleGAN for photo-to-caricature style conversion by swapping fine layers of the StyleGAN for photos to the corresponding layers of the StyleGAN trained to generate caricatures. Given an input photo, the layer-mixed model produces detailed color stylization for a caricature but without shape exaggerations. We then append shape exaggeration blocks to the coarse layers of the layer-mixed model and train the blocks to create shape exaggerations while preserving the characteristic appearances of the input.

Requirements

βœ”οΈ PyTorch 1.3.1
βœ”οΈ torchvision 0.4.2
βœ”οΈ CUDA 10.1/10.2
βœ”οΈ dlib 19.22.0
βœ”οΈ requests 2.23.0
βœ”οΈ tqdm 4.46.2

⚠️ If you are using Anaconda environment and get errors regarding compiler version mismatch, check issue #1.

Usage

First download pre-trained model weights:

bash ./download.sh

Train

python -m torch.distributed.launch --nproc_per_node=N_GPU train.py --name EXPERIMENT_NAME --freeze_D

Test

Test on user's input images:

python test.py --ckpt CHECKPOINT_PATH --input_dir INPUT_IMAGE_PATH --output_dir OUTPUT_CARICATURE_PATH --invert_images

We provide some sample images. Test on sample images:

python test.py --ckpt CHECKPOINT_PATH --input_dir examples/samples --output_dir examples/results --invert_images

It inverts latent codes from input photos and generates caricatures from latent codes.

Examples

Input image Output caricature
img1 cari1
img2 cari2
img3 cari3
img4 cari4

Citation

If you find this code useful, please consider citing:

@article{Jang2021StyleCari,
  author    = {Wonjong Jang and Gwangjin Ju and Yucheol Jung and Jiaolong Yang and Xin Tong and Seungyong Lee},
  title     = {StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation},
  booktitle = {ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH)},
  publisher = {ACM},
  volume = {40},
  number = {4},
  year = {2021}
}

Download pre-trained models

🏷️ StyleCariGAN
🏷️ Photo-StyleGAN (generator_ffhq.pt)
🏷️ Caricature-StyleGAN (generator_cari.pt)
🏷️ Photo-Attribute-Classifier (photo_resnet.pth)
🏷️ Cari-Attribute-Classifier (cari_resnet.pth)

Contact

πŸ“« You can have contact with [email protected] or [email protected]

License

This software is being made available under the terms in the LICENSE file.

Any exemptions to these terms require a license from the Pohang University of Science and Technology.

Credits

❀️ Our code is based on the official StyleGAN2 implementation and rosinality's StyleGAN2-pytorch code
❀️ Specially thanks to CJWBW who ported our project to Replicate.

stylecarigan's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

stylecarigan's Issues

Number of images generated

Hi I have tried the model, really cool!
Just wondering whether it always generate 64 images? is there a way to output the 'best one'/ 'best ones'?
Or do the 64 output from the style palette have style names if I just want to have one of a few output?

Thank you

How to generate caricatures from images

I dont understand how am i suppose to generate caricature from input photos.
I downloaded the checkpoint "001000.pt"

When I run
test.py --invert_images --ckpt checkpoint/StyleCariGAN/001000.pt

I get the error: RuntimeError: File samples/samples\jeniffer256.pt cannot be opened.

Am I somehow suppose to generate .pt files for each of my Images.

download.sh is not working

connect to the link failed,
https:host not found
unlink:no such file or directory
no matched on pattern '120000.pt?dl=0'

Error occurs when testing on my aligned face images!!!

I meet this mistake when I try your test.py by our images.
image
I guess if it because we don't have the args.image_name+'.pt' in our images folder.
and it occurs
image
by the way, our environment is suitable for this project. running on 2080ti + cuda10.1 + torch 1.5.1

Need I train a model myself or where could I get the pt for each images in my folder.

Attribute classifiers

How to train the attribute classifiers?
Where is the training code?
Thank you.

Is InterFaceGAN;s face editing is prior than styleCariGAN in paper?

Firstly, this work is damn fascinating! Admiring you all! Thanks!

I am a beginner of GAN, and I wonder know a naive question, Is InterFaceGAN's face editing is prior than styleCariGAN in paper?

For example, if I understand correctly, I want to draw an open-mouth face caricature from a close-mouth photo, I need to edit face firstly using some GAN like InterFaceGAN, and then use the prior output image, send as input to styleCariGAN?

I test styleCariGAN's xx.pt invert codes seems not compatible with InterFaceGAN's xx.npy format, couldn't edit after style-cari processing directly. haha

colab

please add a google colab for inference

The code fails to run under Anaconda environment

! WARNING !!

Environment

Anaconda environment under Arch Linux.
Python = 3.7 with other requirements with matching versions specified in the README.
(Pytorch=1.3.1)

Steps I took

Run the test.py with

python test.py --ckpt checkpoint/StyleCariGAN/001000.pt --input_dir examples/samples --output_dir examples/results --invert_images

Expectation

The code should run without error

What I get

The code fails to run and I get this error message.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Your compiler (c++) is not compatible with the compiler Pytorch was
built with for this platform, which is g++ on linux. Please
use g++ to to compile your extension. Alternatively, you may
compile PyTorch from source using c++, and then you can also use
c++ to compile your extension.

See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help
with compiling PyTorch from source.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

                          !! WARNING !!

platform=sys.platform))
Traceback (most recent call last):
File "test.py", line 10, in
from invert import *
File "/home/ycjung/Downloads/git/StyleCariGAN/invert.py", line 12, in
from exaggeration_model import StyleCariGAN
File "/home/ycjung/Downloads/git/StyleCariGAN/exaggeration_model.py", line 7, in
from model import PixelNorm, EqualLinear, ConstantInput, StyledConv, ToRGB, ExaggerationLayer
File "/home/ycjung/Downloads/git/StyleCariGAN/model.py", line 13, in
from op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d
File "/home/ycjung/Downloads/git/StyleCariGAN/op/init.py", line 1, in
from .fused_act import FusedLeakyReLU, fused_leaky_relu
File "/home/ycjung/Downloads/git/StyleCariGAN/op/fused_act.py", line 15, in
os.path.join(module_path, "fused_bias_act_kernel.cu"),
File "/home/ycjung/.conda/envs/stylecarigan/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 661, in load
is_python_module)
File "/home/ycjung/.conda/envs/stylecarigan/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 841, in _jit_compile
return _import_module_from_library(name, build_directory, is_python_module)
File "/home/ycjung/.conda/envs/stylecarigan/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1052, in _import_module_from_library
return imp.load_module(module_name, file, path, description)
File "/home/ycjung/.conda/envs/stylecarigan/lib/python3.7/imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "/home/ycjung/.conda/envs/stylecarigan/lib/python3.7/imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: /home/ycjung/.conda/envs/stylecarigan/lib/python3.7/site-packages/torch/../../../libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /tmp/torch_extensions/fused/fused.so)

Analysis

The testing code internally invokes g++ and compiles external modules in op/, which contains several cpp and cu files. The invoked g++ is installed on your system and is probably new (In my case it was 11.1.0). The g++ used for compiling PyTorch 1.3.1 in anaconda environment is old and does not produce binaries compatible with the external module compiled with the new g++.

Troubleshooting

It seems libtorch.so in pytorch/pytorch package is compiled with GCC 7.3.1. So install GCC 7.3.0 in your anaconda environment. (7.3.1 is not in anaconda)

conda install -c anaconda gxx_linux-64=7.3.0

Then run the code with environment variables specifying GCC compilers installed in the conda environment

CC=x86_64-conda_cos6-linux-gnu-gcc CXX=x86_64-conda_cos6-linux-gnu-g++ python test.py --ckpt checkpoint/StyleCariGAN/001000.pt --input_dir examples/custom --output_dir examples/results --invert_images

This solution got the code running fine. I hope it does to your environment!

Train Time

According to the configuration in your paper, how long does it take to train by default? emmmm, can the 24g 3090 be trained?

Update need and add Windows compatibility

Hello,
Your code need updat here:

  • in model.py, line 19:
    old code:
    self.resnet = resnet18(pretrained=True)
    new code:
    self.resnet = resnet18(weights='ResNet18_Weights.DEFAULT')

  • in invert.py, line 19:
    old code:
    perceptual = torchvision.models.vgg16(pretrained=True)
    new code:
    perceptual = torchvision.models.vgg16(weights='VGG16_Weights.DEFAULT')

And your code not Windows compatible. Need install Visual Studio Community edition and add PATH this line (where have cl.exe):

  • for Windows x64: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\bin\Hostx64\x64
  • for Windows x86: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\bin\Hostx64\x86

I then created a testimages folder with an in and an out subfolder. I put in a JPG image and ran the test. This is what I got:
J:\StyleCariGAN-master>test.py --ckpt checkpoint\StyleCariGAN\001000.pt --input_dir testimages\in --output_dir testimages\out --invert_images optimizing w optimizing w: loss_pixel: 0.4483; loss_feature: 4.0835: 100%|β–ˆ| 250/250 [00:23< optimizing wp optimizing wp: loss_pixel: 0.2393; loss_feature: 2.2942: 100%|β–ˆ| 2000/2000 [04: Traceback (most recent call last): File "J:\StyleCariGAN-master\test.py", line 120, in <module> invert(g_ema.photo_generator, perceptual, photo, device, args) File "J:\StyleCariGAN-master\invert.py", line 341, in invert torch.save(result, args.result_dir + f'/{args.image_name}.pt') File "C:\Users\Mykee\miniconda3\lib\site-packages\torch\serialization.py", line 376, in save with _open_file_like(f, 'wb') as opened_file: File "C:\Users\Mykee\miniconda3\lib\site-packages\torch\serialization.py", line 230, in _open_file_like return _open_file(name_or_buffer, mode) File "C:\Users\Mykee\miniconda3\lib\site-packages\torch\serialization.py", line 211, in __init__ super(_open_file, self).__init__(open(name, mode)) FileNotFoundError: [Errno 2] No such file or directory: 'testimages\\in/testimages\\in\\20200913_124752.pt'

The .pt file(s) has not been generated or saved, so it cannot be found. How can I fix this?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.