Contrib: 3D Face Models - Just make your own about facesynthetics HOT 2 OPEN

I think the research team and Microsoft have worked really hard on this research project. Sharing the dataset in it's format and scale is incredibly generous and a good-faith contribution to the ML research community.

So hats off to the team, this is a really exciting paradigm shift not just for facial detection models!

Anyway, for everyone who keeps asking for them to share their 3D assets - they literally tell you everything you need to do it yourself in their short summary video!

I won't write a tutorial but here's a very rough gist of a workflow to create your own 3D face models:

• Read the research abstract and watch the announcement video again. Pay close attention to the brief run through of their 3D model process.
• Create a simple abstract brief of the type of dataset you want to end up with using your own 3D face models
• Work backwards using your understanding of how the research team described their 3D workflow
• From there you can create a rough plan of everything you'll need to collect and learn to replicate their methods.
• If you put some time to figure out the above - Everything below will fill in the blanks and hopefully get excited to learn, experiment and relish in taking the challenge head on.
• Download Blender (It's free and there's plenty of quality tutorials on youtube)
• Download a pre-made 3D generic head model on Sketchfab (Also free)
• You can also find 3d models for facial apparel and hair on Sketchfab
• Get your head around the basics of modelling and rigging (see: Youtube) so that you can iterate head model variants that you can imperatively mutate dynamic poses and facial expressions
• Use a face tracking app on your phone and collect facial expression and head movement data
• Bind your tracking data to your rigged head models iteratively with some clever blender scripting OR
• You can also use Unity or Unreal Engine to programatically apply your tracking data. This will help with render performance and dynamic iteration of Face variants with your 3D facial feature and accessory assets.
  This will cost you in render quality but you've got thousands of face models to render and you'll see from the supplied dataset that your renders aren't expected to be 100% photo realistic.
• I would recommend directing your effort into optimising the anatomical accuracy of your face models.
• Once you bake the tracking into your model variations, you can continue in your game engine context or jump back to blender to add your textures, environment, lighting and animate the camera rotation
• The environment and lighting is a piece of cake - just grab some HDRI's from Polyhaven and apply them to your face model variation scene as well as making sure they emit light - subtle generic scene lighting will help too
• Polyhaven has you covered for your face textures too - be sure to play around with the material settings to take advantage of your lighting, enable shadows and maybe some subsurface scattering for the skin
• Finally create your camera rotation animation and hopefully by this stage you'll have created a little script to iterate through your face models and render your dataset programatically (You can write some simple python code in blender to get that working)
• Oh don't forget to use the Cycles renderer - that's what these guys used

There you go, be free and don't forget to share your 3D assets 😂 Btw, those renders are either going to take weeks to months to finish. Either that or you'll max out your credit card on render farms...

Happy to help anyone brave enough to give this a go!

Edit: extra ideas

Thank you so much for sharing this workflow!

from facesynthetics.

Thanks for this! Was wondering how one would go about annotating landmarks? Would they have to be manually matched to vertices in the rendered 3D mesh?

from facesynthetics.