VTuber-MomoseHiyori

VTuber Demo

Watch MP4 DEMO ( I use a mirror camera so that my behavior is opposite to Hiyori's )
Test Behavior : Nod, Shake, Rotation, Eyeball Rotation, Blink, Eye Half-opening, Mouth Opening

Where does the idea come from ?

Recently, I do some studies on Deep Learning and Computer Vision. At the same time, I realize that I can make a VTuber model by Unity which could simulate my facial expression via computer vision. After watching some tutorials I have made a fantastic Live2D model Momose Hhiyori, and becomes a VTuber successfully !

Development Environment

Test System : Windows 10 64bits
Camera : Integrated Webcam
Socket Transmission : Intranet
Model Made : Live2D Cubism Editor 4.0
Engine : Unity
Script Language : C#
Recognition Algorithm : Deep Learning
Language : Python 3.7 Anaconda
Main Required Library : opencv, dlib, numpy, torch

File Explanation

File	Explanation
Recognition	Packed Algorithm for facial recognition
UnityAssets	Tutorial materials for those want to make Live2D VTuber by self
Hiyori酱~	Starter, quick mode to start program

How to be a VTuber ?

Download and unzip ZIP source file

Install required python libraries ( recommend Anaconda )

I do not test at other operating system, if your OS is not Windows, you'd better test it by yourself

Windows

There are some libraries that I use, you can use pip install -r requirements.txt to install as you like

CPU ( recommend for testing )

Libraries Installation by pip install -r requirements_cpu.txt

Open Anaconda Prompt to install dlib by conda install -c menpo dlib if it doesn't work

GPU

Firstly, please check the your CUDA version : 9.0 / 10.1 / 10.2 / None

Install pytorch by running corresponding command such as conda install pytorch torchvision cudatoolkit=10.2 -c pytorch for 10.2

Install other libraries by pip install -r requirements_gpu.txt.

If you have CUDA 10, pip install onnxruntime-gpu to get faster inference speed using onnx model.

Download VTuber_Hiyori.zip and ckpts.zip ( If you want to use onnxruntime to get faster speed ) at Release

Unzip ckpts and put it under Recognition\face_alignment

Unzip VTuber_Hiyori and start VTuber_MomoseHiyori.exe ( Please wait and do not start any other applications simultaneously !!! )

Run Hiyori酱~.bat

If ひよりちゃん start to simulate your facial expression, congratulations! You have been a VTuber now!

The latest verion has been released, you can download and use them.

Tips

If you find it doesn't recognize well, please try again as following :

Use brighter light : To make your face more clearly, using both natural light and point light seems perfect.
Adjust your position : You can start a camera demo to help you know your position by adding --debug at Hiyori酱~.bat. Run again, let the outer green boundary be larger and central but not larger than demo boundary.
Do not wear glasses : Glasses probably influence on the accuracy of eye recognition.
Show your forehead : Probably your hair is too long to have side effects on recognition of your eyes.

Optimization

Use Live2D instead of 3D model
Add 2 eye events : Eye Half-opening and Eyeball Rotation
Optimize some parameters and be more accurate
Easy start and fixed window at top without boundary, more convenient for live streaming

UnityAssets Tutorial

( If you don't want to know how to import a Live2D VTuber, you can skip this part )

Description : It is a template for most Cubism Live2D models. If you just want to customize your own Live2D models, probably you can read this tutorial and following steps.
Recommend Unity Engine : Unity 2019.4.1f1 LTS
Before you start : Equip yourselves with knowledge of Unity basic operation
Prepare Live2D SDK : You must download SDK on website, or use CubismSdkForUnity-4-r.1 I download for you under UnityAssets
Create a new Unity project

Import Live2D SDK : Drag CubismSdkForUnity-4-r.1 to Assets and choose to import all

Restart Unity : Do not forget this step, otherwise the SDK probably cannot work !
Import Assets : Delete the default scene file, drag Momose, Scece and Script file under Assets

Import Model : A prefab will be automatically generated at Assets/Momose/hiyori_pro_t08.prefab. Open Scene/MomoseHiyori and drag prefab into scene