Comments (5)
Yesterday the question would have been that it's near real-time (couldn't get the data in real-time from OpenFace), but the help of a professor in my lab, we almost got real-time to work (probably today it works ^_^): OpenFace issue about real-time.
Technically lip sync should work. The Facial Action Coding System (FACS) also includes the muscles of the mouth. When I do a promotion video for FACSvatar v0.3, I'll include a speaking person.
The data still has to go through the Python scripts, because a smoothing and a FACS --> Blend Shape conversion is taking place here. Also, if you want to improve the lip-sync, you might want to do some post processing or audio-->FACS mix with the original data. So it's also handy to do this in modules outside a game-engine.
Currently, I don't have time to also support Unreal Engine, but since that engine is offering Python support with its latest release, that integration shouldn't be too hard. Of course, ZeroMQ is not limited to Python, C++ would also work. So, if you want to use that engine, please write some ZeroMQ subscriber code in that engine, would love it to include it in this project :)
from facsvatar.
Hi NumesSanguis
Unfortunately I'm unable to commit to writing code for Unreal. It does make sense though. I've read about your modular approach and was thinking about the various (open source or free for devs) output engines that might be swapped in for this project (and also get this project some more attention from a wider user base).
Blender may also be in the list. The next 2.8 version will have real time rendering with Eevee.
DAZ Studio might also be a possibility - and though a (near)realtime interface might be problematic using your project to create animation curves might be desirable for some.
Presently I can understand 2 use cases for the engine side of things - user needs realtime or user records and possibly edits animation data from open face.
Also head shapes are potentially another module - realistic heads like ManualBastioniLab (and DAZ) and then possibly cartoon heads like this one from Blender user Pepe. (https://vimeo.com/190801903) (though I am not sure of the shapekey/bone/driver setup there)
I've been following BlenderSushi's iPhoneX project for facial mocap. It seems to me that the requirement for cartoons is to radically emphasize emotions. I was wondering if one could detect the rate of change in the OpenFace stream on particular AUs and then emphasize the blendshape with the FACS dictionary.
The JALI project below has an interesting taxonomy of emotions and associated Jaw/Lip curves.
This project (http://www.dgp.toronto.edu/~elf/jali.html) also uses the FACS standard for lip syncing to audio. It is not realtime however because it processes the audio offline.
However if FACSvatar can run in real time then the need to process audio to extract phoneme and visemes is overcome.
Sorry to blabber on :)
This is a very exciting time for facial motion capture!
Cheers
from facsvatar.
Hey daz6549,
I was wondering if you were thinking of a different game engine than Unity 3D, so no worries you cannot do that. Just produce the code you need for your project, share it, and FACSvatar will become more and more useful :)
Streaming into Blender is already working! I think I just wrote the code ugly, because waiting for messages now happens in the main loop of Blender. If I move this to a thread, I think real-time streaming into Blender should work. It works good enough for now for non-realtime: https://www.youtube.com/watch?v=ImB3it_26bc
I'm also excited for Eevee in 2.8, although that will still take a while right?
For exaggerating the facial expressions, other people seems to share the same idea. Yes, you can, here are more details: https://blenderartists.org/forum/showthread.php?391401-Addon-Manuel-Bastioni-Lab-turns-Blender-in-a-laboratory-for-3d-humanoids-creation/page64
If you modify the AU data, the Blend Shapes / Shape Keys are adjusted accordingly.
The JALI project is also great. Hopefully we can reach similar levels with an Open Source implementation.
It's good to see so much excitement ^_^
from facsvatar.
Apologies for responding on a closed issue, but I am curious if FACSvatar is able to extract visemes from audio or if it doesn't include audio in its processing step at all?
I am responding because I was working on a similar project before I came across FACsvatar. I am also using OpenFace and ZeroMQ in a similar fashion as you are but I am also attempting on including a real-time viseme detector for lip-syncing.
I was thinking it might be better to contribute to this project if you were open to the idea (and if viseme/lip-sync real-time detection wasn't already implemented in this project)
from facsvatar.
Hello @lightnarcissus ,
Great we reached a similar idea of how to integrate OpenFace in a project using ZeroMQ!
FACSvatar at present uses only FACS data, but it is designed in a way that you can use other data sources as well.
The idea is that you make stand-alone modules (subscribes to input, publishes output) and link these different modules in a way that make sense to you.
Following this paradigm, you could do something such as:
- Grab audio input from a microphone and publish the data. See the Voice Activity Detector (VAD) module (note: for me it only worked on Windows, but maybe another library, possibly non-python, works better): https://github.com/NumesSanguis/FACSvatar/blob/master/modules/input_vad/pub_vad.py
- Optional module: Take audio input and filter it for speech / non-speech
- Create a module that transforms audio segments into viseme JSON data. ManuelBastioniLAB Human characters come with phoneme facial expressions (https://github.com/animate1978/MB-Lab/tree/master/data/expressions_comb/human_expressions). For an example on how to use those JSON files, see the FACStoBlend module: https://github.com/NumesSanguis/FACSvatar/tree/master/modules/process_facstoblend
- Modify Unity3D code to receive viseme data or write a script for Unreal Engine.
Some notes:
- The original author of the ManuelBastioniLAB add-on for Blender disappeared. A community has been formed around MB-Lab (https://github.com/animate1978/MB-Lab/wiki/Links). A new version of FACSvatar will rely on this branch. In the community are users who have used MB-Lab characters in both Unity3D and UE4. The Discord server contains a channel #face-lip-sync. Probably you can find a lot of useful information there. I'm active on that channel as well.
- Demo video FACSvatar: https://www.youtube.com/watch?v=J2FvrIl-ypU
- A video to help you start: https://www.youtube.com/watch?v=OOoXDfkn8fk
- Note that I'm planning to make the Blender workflow easier by creating a FACSvatar add-on.
from facsvatar.
Related Issues (20)
- How to enhance AUs from within Facsvatar? (Fix the notebook GUI) HOT 9
- Will there be a windows stand-alone for Blender? HOT 2
- Blender avatar is not receiving data. HOT 1
- Problem with face bones HOT 3
- openface_2.1.0_zeromq generated .csv data incompatibility with input_facsfromcsv HOT 5
- How to use FACSvatar system with 2 or more computers
- How to read csv output by OpenFaceOffline ? HOT 13
- Can I fix fps of OpenFaceoffline ? HOT 2
- Openface License Information. No Commercial Use Without Purchase, $18k Per Year HOT 20
- How avatar's setting in Make Human (FACS Human) HOT 2
- Facsdnnfacs for dynamic AUs? HOT 2
- How should I get python output? HOT 2
- Issue with changing bone.rotation_mode to 'XYZ' HOT 4
- Background rotation in visualizing pose angles HOT 6
- Use Google's project MediaPipe Face Mesh as FACS tracker HOT 18
- Character model replacement HOT 1
- How can I save the data captured by camera through OpenFaceOffline into a csv file? HOT 1
- FACSvatar with Blender 3.3 HOT 1
- No json file found for AU07 HOT 1
- Feature: Explore using Reallusion - free 3D Character Base as human model
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from facsvatar.