soubhiksanyal / ringnet Goto Github PK
View Code? Open in Web Editor NEWLearning to Regress 3D Face Shape and Expression from an Image without 3D Supervision
Home Page: https://ringnet.is.tue.mpg.de
License: MIT License
Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision
Home Page: https://ringnet.is.tue.mpg.de
License: MIT License
How to add UV to mesh?
@soubhiksanyal Hi Soubhik, Thanks for your great work. The quantitative evaluation of your model is a little bit vague. Based on what you reported, you computed the error using the method in this paper, which
computes the distance from ground truth scan points to the estimated mesh surface. I went through that paper but computing the scan-to-mesh distance between the predicted meshes and the
scans is still unclear to me. Would you please share the code you used for the evaluation?
Appreciate your time.
run the following command:
python -m demo --img_path ./input_images/000013.jpg --out_folder ./RingNet_output --save_obj_file=True --save_flame_parameters=True --neutralize_expression=True
If I input a face image and generate the input of voca through the above command, is there any way to save the 3d information of the input image and rewrite it into the output obj file of voca, yes, the output of voca Is there any texture information about the input image? That is to say, the output of voca is an obj file with 3d model information (can contain v, f, vn, vf, etc.), mtl file, texture png. Thanks.
hello ,Great work! Will you open source the train code ?
for reconstruction accuracy ,How to crop the face for network inout image?
hello, i run the demo.py , but the image ( generated obj file) in mesh folder is downward. how can i make it upward?
Given aligned(to 2d image) shape model (like here https://raw.githubusercontent.com/soubhiksanyal/RingNet/master/gif/celeba_reconstruction.gif ) is it possible to project face texture from image to 3d model?
Hi, I tried to run the demo on RTX2060 6G, unfortunately ran out of gpu memory, so I want to know what is the minimum GPU memory requirement?
GPU:0 with 591 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:09:00.0, compute capability: 7.5)
Restoring checkpoint ./model/ring_6_68641..
218
Resizing so the max image size is 224..
2020-11-13 22:02:41.101218: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 530.16MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-11-13 22:02:41.101256: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 530.16MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-11-13 22:02:41.117759: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 536.34MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-11-13 22:02:41.117785: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 536.34MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-11-13 22:02:41.123718: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.06GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-11-13 22:02:41.123733: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.06GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-11-13 22:02:41.125294: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 274.25MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-11-13 22:02:41.125308: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 274.25MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-11-13 22:02:41.135688: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 545.31MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-11-13 22:02:41.135703: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 545.31MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
can you tell me how to install psbody packages?
hello,
thanks for your great works!
i want to follow your work, but i can't download the pretrained RingNet weights you provided on your project website. I've signed up, but the download is very slow and always broken, Can you give me some advice?
thank you again!
python -m demo --img_path ./input_images/000013.jpg --out_folder ./RingNet_output --save_obj_file=True --save_flame_parameters=True --neutralize_expression=True
when I run this command its not giving the output of neutral_expression , I have copied the flame models to the flame_model folder. It has given at line 80 in serialization.py "dd = pickle.load(open(fname_or_dict))"
Pls help me in solving this issue and I am using pyton 3.6 as mesh libraries are working only in python 3.5+
Can you please let me know whether I can run code on CPU?
https://github.com/soubhiksanyal/RingNet/blob/master/util/renderer.py
def get_original(proc_param, verts, cam, img_size):
img_size = proc_param['img_size']
...
In the current code, img_size argument is overridden by the proc_param['img_size'.]
Wondering if it's always the case that 'img_size' stored in proc_param should be used or it can actually be passed from the argument.
Greetings.
RingNet output a 159-dimensional vector. which contains 100-dimensional shape parameters, 50-dimension expression parameter and 6-dimensional pose parameter.
However, in Flame, the shape parameter is in shape (300,), expression parameter is in shape (100,), pose parameter is in shape (12,)
If you can give some explanation, I'd be most grateful.
Thanks in advance!
I am wondering, during training, the 2D feature loss requires to project 3D feature points onto 2D plain. How is it done using the 1x3 vector of the predicted camera parameter? Any training code for the project? This is also important if one wants to align the output 3D mesh with the input 2D image.
Hi, I would like to ask how to detect the 68 landmarks which PRNet/3DDFA does.
Thanks in advance.
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory
Failed to load the native TensorFlow runtime.
Being a beginner in this domain, I seek some help to run the code. Thanks in advance!
Hi,
Thank you so much for your great work!
I have an issue. When I run the following command:
python -m demo --img_path ./input_images/000001.jpg --out_folder ./RingNet_output --save_obj_file=True
Output file does not show any mesh result in 2D images like below:
On the other hand, it produces the mesh like below:
opendr version is 0.77.
I would like to ask how can I get the predicted (output of the model) vertex coordinates or landmark coordinates (points) of the original 2D input image?
Hi, thanks for this amazing work! I am trying to recover the mesh from the predicted parameters(100 D shape, 50 expression and 6 dimensional pose), and I used a modified version "make_prdicted_mesh_neutral' function(copy flame-fitting project) to do this, it seems the shape and pose is correct, but the expression is wrong. Am I missed something? Thanks a lot.
`
def make_prdicted_mesh_neutral_ringnet(predicted_params_path, flame_model_path):
params = np.load(predicted_params_path, allow_pickle=True)
params = params[()]
pose,betas = np.zeros(15),np.zeros(400)
pose[:3],pose[-3:]= params['pose'][:3],params['pose'][3:]
betas[:params['shape'].shape[0]] = params['shape']
betas[300:300+params['expression'].shape[0]] = params['expression']
flame_genral_model = load_model(flame_model_path)
generated_neutral_mesh = verts_decorated(ch.array([0,0,0]),
ch.array(pose),
ch.array(flame_genral_model.r),
flame_genral_model.J_regressor,
ch.array(flame_genral_model.weights),
flame_genral_model.kintree_table,
flame_genral_model.bs_style,
flame_genral_model.f,
bs_type=flame_genral_model.bs_type,
posedirs=ch.array(flame_genral_model.posedirs),
betas=ch.array(betas),#betas=ch.array(np.concatenate((theta[0,75:85], np.zeros(390)))), #
shapedirs=ch.array(flame_genral_model.shapedirs),
want_Jtr=True)
return generated_neutral_mesh
is it possible to preserve identity for the output meshes, in the gif added to the readme it shows a stage where the identities are preserved and later generalized, can the output obj file be set to the previous stage where identity is preserved?
Thank you for your sharing.
I want to run it on windows, but the opendr file has not been successfully installed.
In your paper ,you use Differenciable FLAME decode model to get 3d mesh by flame parameters, and this FLAME decode model is fixed during training .My question is How to get this Differenciable FLAME decode model?
@soubhiksanyal I tested RingNet with some images extracted from RAVDESS. It seems RingNet can't work well for closed eyes, just as the following image shows.
Is this because you just used 50 expression blendshapes, or because the limitation of the training data, or others?
Hi~
Thank you so much for your amazing work!
I originally wanted to use the following code to remove the global rotation of the generated model, but because of the limited expressiveness of the parameters, the results are different from the original model.
I would like to ask how to calculate directly on the vertex coordinates of the original model?
params = np.load(out_param, allow_pickle=True)
params = params[()]
#pose = np.hstack((params['pose'], np.zeros(15-params['pose'].shape[0])))
pose = np.zeros(15)
pose[6] = params['pose'][3]
pose[7] = params['pose'][4]
pose[8] = params['pose'][5]
expression = np.hstack((params['expression'], np.zeros(100-params['expression'].shape[0])))
shape = np.hstack((params['shape'], np.zeros(300-params['shape'].shape[0])))
flame_genral_model = load_model(config.flame_model_path)
generated_neutral_mesh = verts_decorated(ch.array([0.0,0.0,0.0]),
ch.array(pose),
ch.array(flame_genral_model.r),
flame_genral_model.J_regressor,
ch.array(flame_genral_model.weights),
flame_genral_model.kintree_table,
flame_genral_model.bs_style,
flame_genral_model.f,
bs_type=flame_genral_model.bs_type,
posedirs=ch.array(flame_genral_model.posedirs),
betas=ch.array(np.hstack((shape,expression))),#betas=ch.array(np.concatenate((theta[0,75:85], np.zeros(390)))), #
shapedirs=ch.array(flame_genral_model.shapedirs),
want_Jtr=True)
neutral_mesh = Mesh(v=generated_neutral_mesh.r, f=generated_neutral_mesh.f)
neutral_mesh.write_obj(out_mesh)
print("Saved neutral mesh file to " + out_mesh)
@soubhiksanyal @TimoBolkart Is there any chance you guys will be uploading the network architecture file soon?
"RingNet outputs a subset of the FLAME parameters. These parameters are the first 100 shape parameters, first 50 expression parameters, and the pose parameters for the global rotation (FLAME pose parameters 1-3) and the jaw rotation (FLAME pose parameters 7-9). "
Originally posted by @TimoBolkart in #7 (comment)
Wondering what are the FLAME pose parameters 4-6. Are these for the neck which is also a joint in the FLAME model? Does RingNet outputs pose parameters for neck?
Thanks,
Bing
I am new to programming and cv in general, so forgive me if my issues seem too simple.
I am using Ubuntu 18.04 and trying to replicate your included demo just to make sure the program is working. I have repeatedly run into different problems that all seem to be related to CUDA and tensorflow.
I have heard that these programs are all very sensitive to driver versions and dependencies. So far, the furthest I seem to get is to a point where I get an absl module missing error. I am sure it is installed, and have tried various version. At one point downgrading forced me to downgrade to tensorflow 1.12. Still the same issue is popping up. I am ready to start over step by step, as tedious as that is.
I wonder if anybody could explain to me how to make sure that the versions of all the needed libraries and whatnot to run RingNet are correct.
I can't find the supplementary materials of this CVPR2019 paper in project website.
When to release it?
Thanks!
Pre-trained models are available in the Downloads page:
https://ringnet.is.tue.mpg.de/downloads
Thanks your excelent work.As python2.7 has be deprecated,I wonder if it is schedled to upgrade Python version for this project.
Hi,
When I run the following command with the downloaded pre-trained model:
python -m demo --img_path ./input_images/000001.jpg --out_folder ./RingNet_output --save_obj_file=True
and I'm seeing the following output file.
output
The pose of the reconstructed mesh is like this :
My opendr version is 0.77
Hi,
I'm running the following command with the downloaded pre-trained model:
python -m demo --img_path ./input_images/000013.jpg --out_folder ./RingNet_output
and I'm seeing the following output file.
The renders are all dark. When I ran the same command with the flag --save_obj_file=True,
the obj file was successfully created and I could view it successfully in Blender.
i can not download model
Do you have any plans to release training code and network architecture file? @soubhiksanyal
I show you updated the demo code yesterday,but i cannot find the description of flame_textrue_data. So how to get flame_textrue_data?
This project requires pyrender which requires unversioned imageio, which no longer supports Python 2.7 since February.
I followed the instructions for cloning and building mesh but when I run demo I get:
Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/home/username/RingNet/demo.py", line 45, in <module> from psbody.mesh import Mesh ImportError: No module named psbody.mesh
How do I tell the RingNet virtual environment to load mesh? I tried running setup install on mesh in the RingNet virtual environment and just got more errors. mesh appears to be python 3 now and RingNet is Python 2 so do I have to use an older version of mesh that is python 2?
Hello, thanks or the release of this project! Which version of OpenCV did you use?
Excellent work for 3D face reconstruction from 2D images!
The showing gif and demo video also exhibit eyeballs gazing directions. However, from paper and experiments, I found that the model just outputs 3D mesh, FLAME pose parameters and the jaw rotation. Thus, can we get the rotation matrix of eyeballs? Or I guess that in the demo, the rotations of eyeballs are the same as the rotations of head? Is my guess right?
Hello, since my computer does not support CUDA 9, so I installed it on Google Colab. But Colab does not support virtualenv, and RingNet requires virtualenv to use Mesh.
Any way to get RingNet run on Colab?
Thank you.
Hi, thanks for the amazing work. I have been doing some research work on expressions, and I decided to use RingNet to extract the expression parameters. However, I don't know how to use these parameters to realize 3D visualization. In addition, to observe the coherence of the expression sequence, I need to obtain the 2D projection of Flame model. Could you do me a favor on these problems?
I downloads ana run your code on my platform follow your instruction, and I have download all the dependence of this project, but I find dispite the demo picture, all the other input will get almost the same result. I think some preprocessing is needed for the input picture, but I cannot find it on your project page or the paper, can you please tell me what preprocessing you do for the input picture? I will thank you more if you can show me the preprocessing code
pretrained models - permission denied
When I tried this repository many of the meshes looks similar with another. Hence the texture is really important for identification. How can I add texture to the meshes for using them in VOCA. I have tried with TF_FLAME for creating texture for the mesh using the image file but it was not usable in VOCA. Can you suggest how can I do this?
During the training, 2D feature loss is used. I am wondering if you can provide indexes of trajectories/candidate contour points? If you can provide the code of 2D feature loss, I'd most grateful.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.