Project page | Video | Paper
Official implementation of DCT-Net for Full-body Portrait Stylization.
DCT-Net: Domain-Calibrated Translation for Portrait Stylization,
Yifang Men1, Yuan Yao1, Miaomiao Cui1, Zhouhui Lian2, Xuansong Xie1,
1DAMO Academy, Alibaba Group, Beijing, China
2Wangxuan Institute of Computer Technology, Peking University, China
In: SIGGRAPH 2022 (TOG) arXiv preprint
(2023-03-14) The training guidance has been released, train DCT-Net with your own style data.
(2023-02-20) Two new style pre-trained models (design, illustration) trained with combined DCT-Net and Stable-Diffusion are provided. The training guidance will be released soon.
(2022-10-09) The multi-style pre-trained models (3d, handdrawn, sketch, artstyle) and usage are available now.
(2022-08-08) The pertained model and infer code of 'anime' style is available now. More styles coming soon.
(2022-08-08) cartoon function can be directly call from pythonSDK.
(2022-07-07) The paper is available now at arxiv(https://arxiv.org/abs/2207.02426).
-
Integrated into Colab notebook. Try out the colab demo.
-
Integrated into Huggingface Spaces 🤗 using Gradio. Try out the Web Demo
-
[Chinese version] Integrated into ModelScope. Try out the Web Demo
- python 3
- tensorflow (>=1.14, training only support tf1.x)
- easydict
- numpy
- both CPU/GPU are supported
git clone https://github.com/menyifang/DCT-Net.git
cd DCT-Net
conda create -n dctnet python=3.7
conda activate dctnet
pip install --upgrade tensorflow-gpu==1.15 # GPU support, use tensorflow for CPU only
pip install "modelscope[cv]==1.3.2" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html
pip install "modelscope[multi-modal]==1.3.2" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html |
 |
 |
 |
 |
|---|---|---|---|---|
| anime | 3d | handdrawn | sketch | artstyle |
 |
 |
|---|---|
| design | illustration |
Pre-trained models in different styles can be downloaded by
python download.py- from python SDK
python run_sdk.py- from source code
python run.py
video can be directly processed as image sequences, style choice [option: anime, 3d, handdrawn, sketch, artstyle, sd-design, sd-illustration]
python run_vid.py --style animeface_photo: face dataset such as [FFHQ](https://github.com/NVlabs/ffhq-dataset) or other collected real faces.
face_cartoon: 100-300 cartoon face images in a specific style, which can be self-collected or synthsized with generative models.
Due to the copyrighe issues, we can not provide collected cartoon exemplar for training. You can produce cartoon exemplars with the style-finetuned Stable-Diffusion (SD) models, which can be downloaded from modelscope or huggingface hubs.
The effects of some style-finetune SD models are as follows:
 |
 |
 |
 |
 |
|---|---|---|---|---|
| design | watercolor | illustration | clipart | flat |
- Generate stylized data, style choice [option: clipart, design, illustration, watercolor, flat]
python generate_data.py --style clipart- preprocess
extract aligned faces from raw style images:
python extract_align_faces.py --src_dir 'data/raw_style_data'- train content calibration network
install environment required by stylegan2-pytorch
cd source/stylegan2
python prepare_data.py '../../data/face_cartoon' --size 256 --out '../../data/stylegan2/traindata'
python train_condition.py --name 'ffhq_style_s256' --path '../../data/stylegan2/traindata' --config config/conf_server_train_condition_shell.jsonafter training, generated content calibrated samples via:
python style_blend.py --name 'ffhq_style_s256'
python generate_blendmodel.py --name 'ffhq_style_s256' --save_dir '../../data/face_cartoon/syn_style_faces'- geometry calibration
run geometry calibration for both photo and cartoon:
cd source
python image_flip_agument_parallel.py --data_dir '../data/face_cartoon'
python image_scale_agument_parallel_flat.py --data_dir '../data/face_cartoon'
python image_rotation_agument_parallel_flat.py --data_dir '../data/face_cartoon'- train texture translator
The dataset structure is recommended as:
+—data
| +—face_photo
| +—face_cartoon
resume training from pretrained model in similar style,
style can be chosen from 'anime, 3d, handdrawn, sketch, artstyle, sd-design, sd-illustration'
python train_localtoon.py --data_dir PATH_TO_YOU_DATA --work_dir PATH_SAVE --style animeFace detector and aligner are adapted from Peppa_Pig_Face_Engine and InsightFace.
If you find this code useful for your research, please use the following BibTeX entry.
@inproceedings{men2022dct,
title={DCT-Net: Domain-Calibrated Translation for Portrait Stylization},
author={Men, Yifang and Yao, Yuan and Cui, Miaomiao and Lian, Zhouhui and Xie, Xuansong},
journal={ACM Transactions on Graphics (TOG)},
volume={41},
number={4},
pages={1--9},
year={2022},
publisher={ACM New York, NY, USA}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent
