zielon / insta Goto Github PK
View Code? Open in Web Editor NEWINSTA - Instant Volumetric Head Avatars [CVPR2023]
Home Page: https://zielon.github.io/insta/
License: Other
INSTA - Instant Volumetric Head Avatars [CVPR2023]
Home Page: https://zielon.github.io/insta/
License: Other
From issue #6, it was mentioned that for expression transfer, we need the new meshes corresponding to the new expressions. How do we obtain the meshes from desired FLAME expressions?
I've seen that length-100 FLAME vectors are used, but I can't get them to affect the expression of the output video.
when I use the demo checkpoint to load the data in WSL ubuntu:
./build/rta --config insta.json --scene data/obama/transforms_test.json --snapshot data/obama/snapshot.msgpack
and there is no GUI show, and where is the person's 3D models or results something, can you help me please?
colorama==0.4.6 commentjson==0.9.0 imageio==2.28.1 lark-parser==0.7.8 numpy==1.23.5 opencv-python==4.7.0.72 Pillow==9.5.0 pybind11==2.10.4 pyquaternion==0.9.9 scipy==1.10.1 tqdm==4.65.0
I followed the installation instructions with a few changes to the Cmakelists.txt, namely, replacing the regular find_package(PythonLibs)
with find_package(Python 3.7 COMPONENTS Interpreter Development) add_subdirectory("${CMAKE_CURRENT_SOURCE_DIR}/instant-ngp/dependencies/pybind11")
I was able to compile the current instant-ngp repo, but not this one.
the error i get is fatal error C1083: Cannot open include file: 'fmt/core.h': No such file or directory
cmake_output.log
I moved past that by reintroducing get_target_property(TCNN_INCLUDE_DIRECTORIES tiny-cuda-nn INCLUDE_DIRECTORIES) target_include_directories(optix_program PUBLIC ${NGP_INCLUDE_DIRECTORIES} ${TCNN_INCLUDE_DIRECTORIES})
into the CMakeLists.txt, but that also led to an error, suggesting an incompatibility with the OptiX library. I noticed that in the instant-ngp repo they manually supplied the optix header files, which seems to avoid this. NVlabs/instant-ngp@c14da84
So is there a strict dependency on a certain version of OptiX for this repo? If so, which one do I install?
Here's the log from immediately after modifying the CMakeLists.txt
cmake_output2.log
and here's the log from subsequent build attempts.
cmake_output3.log
I have a probably-basic question :)
If I understand how INSTA is working correctly, then all the data used for training and inference is in the following files / array:
"depth_path": "depth/00002.png",
"exp_path": "flame/exp/00002.txt",
"eyes_path": "flame/eyes/00002.txt",
"file_path": "images/00002.png",
"mesh_path": "meshes/00002.obj",
"seg_mask_path": "seg_mask/00002.png",
"transform_matrix": [...]
My question is - where in this data are the FLAME parameters for the neck and jaw joints? I would think if anything they are contained implicitly in the mesh, but when I open the meshes they seem to always be in the same pose. What am I missing here? Thanks!
Where is the final MP4 video saved?
Hi,
since I have only a GTX 3080Ti, I have tried to load the snapshot. But unfortunately it crashes since it cannot allocate enough memory. None of the cli options does what I want. The Readme says: ./build/rta --config insta.json --scene data/obama --height 1024 --width 1024 --snapshot data/obama/snapshot.msgpack
But if I provide the --scene option it tries to load the meshes onto the gpu which makes it crash. When I omit this option, it wants the --mode option. I tried to specify --mode=nerf, but then it crashes again with. For --mode=volume is says Network config path insta.json does not exist.
"ERROR Uncaught exception: Can't set params because CPU buffer has the wrong size."
I think something is missing to only test a rendering example.
Thx
Janusch
It's a great work.
May I know the exact FPS for inference on average?
CMake Warning at CMakeLists.txt:213 (message):
OptiX was not found. Neural graphics primitives will still compile and run
correctly. However, SDF training in 'raystab' and 'pathescape' modes will
be significantly slower.
Hi, thanks for the great work and amazing code! I have a small question -- it seems when loading the expression from txt file, only first 16 digits are kept. May I ask what are those first 16 digits, and why this is done?
Many thanks!
I have the following warning when I run it, but it works, will it affect the effect?
Warning: FullyFusedMLP is not supported for the selected architecture 70. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Can it be voice driven? Can be combined with something like faceformer
Thanks for you work. How to save rendered rgba/normals/depth image with --no-gui option enabled. I dont see any save functions in main train/render loop.
Hi, Thanks for sharing the excellent work!
Now, I am going to perform a face reenactment. The provided datasets contain the 3DMM expression coefficient and camera pose of each frame. However, the 3DMM shape coefficient is not provided for each dataset. I use the MICA
in README.md to estimate the shape coefficient, but the results are not satisfied since there is always a bias toward the canonical.obj
you provided.
Therefore, I am wondering if you could provide the 3DMM shape coefficient identity.npy
of each dataset (not for each frame) ?
When I run the second step, the program keeps running and cannot be stopped. My command is
./build/rta --config insta.json --scene data/obama/transforms_test.json --snapshot data/obama/snapshot.msgpack --no-gui
Hi, hope you can provide some advices. Thanks very much.
Hi,
I made the data with just the head and neck and it ran successfully, but how do I go further and make the data with the shoulders? Only need to modify the mask in postprocess.py?
best,
chieh
Hey @Zielon
Congrats on the great work and thank you for releasing the code.
I am trying to use INSTA on data captured in a light stage setting. I was able to successfully run MICA on the dataset however metrical-tracker does not perform well on the data because of the obvious reasons. What would be the best way to adopt INSTA to date captured using a light stage system? Thank you.
Regards,
Pradyumna.
Hello,
First of all, I would like to commend you on the excellent work you have done. It has provided valuable insights for related research.
While processing my own data using the face-parsing.PyTorch library (https://github.com/zllrunning/face-parsing.PyTorch), I noticed that the parsing results differed from your given examples. In the source code, a face is divided into multiple colors, creating a multicolored representation. You can see an example of this below:
However, the given examples showcase parsing colors that primarily consist of different levels of red, as shown in the following image:
This leads me to inquire about the new part_colors variable in the face-parsing.PyTorch/test.py script of your project. I believe it plays a significant role in achieving the final optimized results.
Thank you for your attention, and I look forward to your response.
Best regards.
Hi.
I'm trying to follow https://github.com/Zielon/INSTA#installation
"C:\Program Files\Git\bin\git.exe" clone -v --recurse-submodules --depth 1 --no-single-branch --progress "https://github.com/Zielon/INSTA.git" "C:/Users/user1/source/repos/Zielon/INSTA"
and facing with error:
PS C:\Users\user1\source\repos\Zielon\INSTA> cmake . -B build
-- Building for: Visual Studio 17 2022
-- Selecting Windows SDK version 10.0.22000.0 to target Windows 10.0.19045.
-- The C compiler identification is MSVC 19.37.32825.0
-- The CXX compiler identification is MSVC 19.37.32825.0
-- The CUDA compiler identification is NVIDIA 12.1.105
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.1/bin/nvcc.exe - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Targeting GPU architectures: 75
-- Module support is disabled.
-- Version: 9.0.0
-- Build type:
-- CXX_STANDARD: 14
-- Required features: cxx_variadic_templates
-- Found Vulkan: C:/VulkanSDK/1.3.268.0/Lib/vulkan-1.lib (found version "1.3.268") found components: glslc glslangValidator
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - not found
-- Found Threads: TRUE
-- Using Win32 for window creation
-- Found OpenMP_C: -openmp (found version "2.0")
-- Found OpenMP_CXX: -openmp (found version "2.0")
-- Found OpenMP: TRUE (found version "2.0")
-- OptiX_INSTALL_DIR value: C:\ProgramData\NVIDIA Corporation\OptiX SDK 8.0.0
-- Found PythonLibs: C:/Python311/libs/python311.lib (found version "3.11.0")
-- Configuring done (10.5s)
-- Generating done (0.1s)
-- Build files have been written to: C:/Users/user1/source/repos/Zielon/INSTA/build
PS C:\Users\user1\source\repos\Zielon\INSTA> cmake --build build --config RelWithDebInfo -j
MSBuild version 17.7.2+d6990bcfa for .NET Framework
Checking Build System
Building Custom Rule C:/Users/user1/source/repos/Zielon/INSTA/CMakeLists.txt
Building Custom Rule C:/Users/user1/source/repos/Zielon/INSTA/instant-ngp/dependencies/glfw/src/CMakeLists.txt
context.c
init.c
input.c
monitor.c
vulkan.c
window.c
win32_init.c
win32_joystick.c
win32_monitor.c
win32_time.c
win32_thread.c
win32_window.c
wgl_context.c
egl_context.c
osmesa_context.c
Generating Code...
glfw_objects.vcxproj -> C:\Users\user1\source\repos\Zielon\INSTA\build\instant-ngp\dependencies\glfw\src\glfw_obje
cts.dir\RelWithDebInfo\glfw_objects.lib
Compiling CUDA source file ..\instant-ngp\src\optix\pathescape.cu...
Compiling CUDA source file ..\instant-ngp\src\optix\raystab.cu...
Compiling CUDA source file ..\instant-ngp\src\optix\raytrace.cu...
C:\Users\user1\source\repos\Zielon\INSTA\build>"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin\nvcc.
exe" --use-local-env -ccbin "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\bin\H
ostX64\x64" -x cu -IC:\VulkanSDK\1.3.268.0\Include -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\depen
dencies\dlss\include" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\glfw\include" -I"C:\Use
rs\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\imgui" -I"C:\Users\user1\source\repos\Zielon\INSTA\
instant-ngp\dependencies\gl3w" -I"C:\ProgramData\NVIDIA Corporation\OptiX SDK 8.0.0\include" -I"C:\Users\user1\sou
rce\repos\Zielon\INSTA\instant-ngp\dependencies" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependenc
ies\tiny-cuda-nn\include" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\tiny-cuda-nn\depend
encies" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\tinylogger" -I"C:\Users\user1\sour
ce\repos\Zielon\INSTA\instant-ngp\dependencies\eigen" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\depe
ndencies\filesystem" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\nanovdb" -IC:\Python311\
include -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\include" -IC:\Users\user1\source\repos\Zielon\I
NSTA\include -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include" --keep-dir x64\RelWithDebInfo
-maxrregcount=0 --machine 64 -ptx -cudart shared -std=c++14 --expt-relaxed-constexpr -Xcompiler="/EHsc -Zi -Ob1" -D
_WINDOWS -DNDEBUG -DNGP_VULKAN -DGLFW_INCLUDE_VULKAN -DNGP_GUI -DNGP_OPTIX -D"NGP_VERSION=\"1.0dev\"" -DTCNN_MIN_GPU_
ARCH=0 -D"CMAKE_INTDIR=\"RelWithDebInfo\"" -D_MBCS -D"CMAKE_INTDIR=\"RelWithDebInfo\"" -o optix_program.dir\RelWithDe
bInfo\pathescape.ptx "C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\src\optix\pathescape.cu"
C:\Users\user1\source\repos\Zielon\INSTA\build>"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin\nvcc.
exe" --use-local-env -ccbin "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\bin\H
ostX64\x64" -x cu -IC:\VulkanSDK\1.3.268.0\Include -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\depen
dencies\dlss\include" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\glfw\include" -I"C:\Use
rs\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\imgui" -I"C:\Users\user1\source\repos\Zielon\INSTA\
instant-ngp\dependencies\gl3w" -I"C:\ProgramData\NVIDIA Corporation\OptiX SDK 8.0.0\include" -I"C:\Users\user1\sou
rce\repos\Zielon\INSTA\instant-ngp\dependencies" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependenc
ies\tiny-cuda-nn\include" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\tiny-cuda-nn\depend
encies" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\tinylogger" -I"C:\Users\user1\sour
ce\repos\Zielon\INSTA\instant-ngp\dependencies\eigen" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\depe
ndencies\filesystem" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\nanovdb" -IC:\Python311\
include -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\include" -IC:\Users\user1\source\repos\Zielon\I
NSTA\include -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include" --keep-dir x64\RelWithDebInfo
-maxrregcount=0 --machine 64 -ptx -cudart shared -std=c++14 --expt-relaxed-constexpr -Xcompiler="/EHsc -Zi -Ob1" -D
_WINDOWS -DNDEBUG -DNGP_VULKAN -DGLFW_INCLUDE_VULKAN -DNGP_GUI -DNGP_OPTIX -D"NGP_VERSION=\"1.0dev\"" -DTCNN_MIN_GPU_
ARCH=0 -D"CMAKE_INTDIR=\"RelWithDebInfo\"" -D_MBCS -D"CMAKE_INTDIR=\"RelWithDebInfo\"" -o optix_program.dir\RelWithDe
bInfo\raytrace.ptx "C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\src\optix\raytrace.cu"
C:\Users\user1\source\repos\Zielon\INSTA\build>"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin\nvcc.
exe" --use-local-env -ccbin "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\bin\H
ostX64\x64" -x cu -IC:\VulkanSDK\1.3.268.0\Include -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\depen
dencies\dlss\include" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\glfw\include" -I"C:\Use
rs\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\imgui" -I"C:\Users\user1\source\repos\Zielon\INSTA\
instant-ngp\dependencies\gl3w" -I"C:\ProgramData\NVIDIA Corporation\OptiX SDK 8.0.0\include" -I"C:\Users\user1\sou
rce\repos\Zielon\INSTA\instant-ngp\dependencies" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependenc
ies\tiny-cuda-nn\include" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\tiny-cuda-nn\depend
encies" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\tinylogger" -I"C:\Users\user1\sour
ce\repos\Zielon\INSTA\instant-ngp\dependencies\eigen" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\depe
ndencies\filesystem" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\nanovdb" -IC:\Python311\
include -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\include" -IC:\Users\user1\source\repos\Zielon\I
NSTA\include -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include" --keep-dir x64\RelWithDebInfo
-maxrregcount=0 --machine 64 -ptx -cudart shared -std=c++14 --expt-relaxed-constexpr -Xcompiler="/EHsc -Zi -Ob1" -D
_WINDOWS -DNDEBUG -DNGP_VULKAN -DGLFW_INCLUDE_VULKAN -DNGP_GUI -DNGP_OPTIX -D"NGP_VERSION=\"1.0dev\"" -DTCNN_MIN_GPU_
ARCH=0 -D"CMAKE_INTDIR=\"RelWithDebInfo\"" -D_MBCS -D"CMAKE_INTDIR=\"RelWithDebInfo\"" -o optix_program.dir\RelWithDe
bInfo\raystab.ptx "C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\src\optix\raystab.cu"
C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/common.h(40): fa
tal error C1083: Cannot open include file: 'fmt/core.h': No such file or directory [C:\Users\user1\source\repos\Ziel
on\INSTA\build\optix_program.vcxproj]
C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/common.h(40): fa
tal error C1083: Cannot open include file: 'fmt/core.h': No such file or directory [C:\Users\user1\source\repos\Ziel
on\INSTA\build\optix_program.vcxproj]
raytrace.cu
pathescape.cu
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\BuildCustomizations\CUDA 12.1.targets
(799,9): error MSB3721: The command ""C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin\nvcc.exe" --use-loc
al-env -ccbin "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64" -x cu
-IC:\VulkanSDK\1.3.268.0\Include -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\dlss\includ
e" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\glfw\include" -I"C:\Users\user1\source\re
pos\Zielon\INSTA\instant-ngp\dependencies\imgui" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencie
s\gl3w" -I"C:\ProgramData\NVIDIA Corporation\OptiX SDK 8.0.0\include" -I"C:\Users\user1\source\repos\Zielon\INSTA\in
stant-ngp\dependencies" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\tiny-cuda-nn\include" -
I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\tiny-cuda-nn\dependencies" -I"C:\Users\user1\
source\repos\Zielon\INSTA\instant-ngp\dependencies\tinylogger" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-n
gp\dependencies\eigen" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\filesystem" -I"C:\Users\
user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\nanovdb" -IC:\Python311\include -I"C:\Users\user1\source
\repos\Zielon\INSTA\instant-ngp\include" -IC:\Users\user1\source\repos\Zielon\INSTA\include -I"C:\Program Files\NVID
IA GPU Computing Toolkit\CUDA\v12.1\include" --keep-dir x64\RelWithDebInfo -maxrregcount=0 --machine 64 -ptx -cu
dart shared -std=c++14 --expt-relaxed-constexpr -Xcompiler="/EHsc -Zi -Ob1" -D_WINDOWS -DNDEBUG -DNGP_VULKAN -DGLFW_INC
LUDE_VULKAN -DNGP_GUI -DNGP_OPTIX -D"NGP_VERSION=\"1.0dev\"" -DTCNN_MIN_GPU_ARCH=0 -D"CMAKE_INTDIR=\"RelWithDebInfo\""
-D_MBCS -D"CMAKE_INTDIR=\"RelWithDebInfo\"" -o optix_program.dir\RelWithDebInfo\raytrace.ptx "C:\Users\user1\source\
repos\Zielon\INSTA\instant-ngp\src\optix\raytrace.cu"" exited with code 2. [C:\Users\user1\source\repos\Zielon\INSTA
\build\optix_program.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\BuildCustomizations\CUDA 12.1.targets
(799,9): error MSB3721: The command ""C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin\nvcc.exe" --use-loc
al-env -ccbin "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64" -x cu
-IC:\VulkanSDK\1.3.268.0\Include -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\dlss\includ
e" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\glfw\include" -I"C:\Users\user1\source\re
pos\Zielon\INSTA\instant-ngp\dependencies\imgui" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencie
s\gl3w" -I"C:\ProgramData\NVIDIA Corporation\OptiX SDK 8.0.0\include" -I"C:\Users\user1\source\repos\Zielon\INSTA\in
stant-ngp\dependencies" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\tiny-cuda-nn\include" -
I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\tiny-cuda-nn\dependencies" -I"C:\Users\user1\
source\repos\Zielon\INSTA\instant-ngp\dependencies\tinylogger" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-n
gp\dependencies\eigen" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\filesystem" -I"C:\Users\
user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\nanovdb" -IC:\Python311\include -I"C:\Users\user1\source
\repos\Zielon\INSTA\instant-ngp\include" -IC:\Users\user1\source\repos\Zielon\INSTA\include -I"C:\Program Files\NVID
IA GPU Computing Toolkit\CUDA\v12.1\include" --keep-dir x64\RelWithDebInfo -maxrregcount=0 --machine 64 -ptx -cu
dart shared -std=c++14 --expt-relaxed-constexpr -Xcompiler="/EHsc -Zi -Ob1" -D_WINDOWS -DNDEBUG -DNGP_VULKAN -DGLFW_INC
LUDE_VULKAN -DNGP_GUI -DNGP_OPTIX -D"NGP_VERSION=\"1.0dev\"" -DTCNN_MIN_GPU_ARCH=0 -D"CMAKE_INTDIR=\"RelWithDebInfo\""
-D_MBCS -D"CMAKE_INTDIR=\"RelWithDebInfo\"" -o optix_program.dir\RelWithDebInfo\pathescape.ptx "C:\Users\user1\sourc
e\repos\Zielon\INSTA\instant-ngp\src\optix\pathescape.cu"" exited with code 2. [C:\Users\user1\source\repos\Zielon\I
NSTA\build\optix_program.vcxproj]
C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/common.h(40): fa
tal error C1083: Cannot open include file: 'fmt/core.h': No such file or directory [C:\Users\user1\source\repos\Ziel
on\INSTA\build\optix_program.vcxproj]
raystab.cu
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\BuildCustomizations\CUDA 12.1.targets
(799,9): error MSB3721: The command ""C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin\nvcc.exe" --use-loc
al-env -ccbin "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64" -x cu
-IC:\VulkanSDK\1.3.268.0\Include -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\dlss\includ
e" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\glfw\include" -I"C:\Users\user1\source\re
pos\Zielon\INSTA\instant-ngp\dependencies\imgui" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencie
s\gl3w" -I"C:\ProgramData\NVIDIA Corporation\OptiX SDK 8.0.0\include" -I"C:\Users\user1\source\repos\Zielon\INSTA\in
stant-ngp\dependencies" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\tiny-cuda-nn\include" -
I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\tiny-cuda-nn\dependencies" -I"C:\Users\user1\
source\repos\Zielon\INSTA\instant-ngp\dependencies\tinylogger" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-n
gp\dependencies\eigen" -I"C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\filesystem" -I"C:\Users\
user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\nanovdb" -IC:\Python311\include -I"C:\Users\user1\source
\repos\Zielon\INSTA\instant-ngp\include" -IC:\Users\user1\source\repos\Zielon\INSTA\include -I"C:\Program Files\NVID
IA GPU Computing Toolkit\CUDA\v12.1\include" --keep-dir x64\RelWithDebInfo -maxrregcount=0 --machine 64 -ptx -cu
dart shared -std=c++14 --expt-relaxed-constexpr -Xcompiler="/EHsc -Zi -Ob1" -D_WINDOWS -DNDEBUG -DNGP_VULKAN -DGLFW_INC
LUDE_VULKAN -DNGP_GUI -DNGP_OPTIX -D"NGP_VERSION=\"1.0dev\"" -DTCNN_MIN_GPU_ARCH=0 -D"CMAKE_INTDIR=\"RelWithDebInfo\""
-D_MBCS -D"CMAKE_INTDIR=\"RelWithDebInfo\"" -o optix_program.dir\RelWithDebInfo\raystab.ptx "C:\Users\user1\source\r
epos\Zielon\INSTA\instant-ngp\src\optix\raystab.cu"" exited with code 2. [C:\Users\user1\source\repos\Zielon\INSTA\b
uild\optix_program.vcxproj]
PS C:\Users\user1\source\repos\Zielon\INSTA>
I also checked this guide and it's repo https://github.com/NVlabs/instant-ngp#compilation, but it builds fine.
Not sure how an import in C works, but I found that file C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/common.h
:
...
#pragma once
// A macro is used such that external tools won't end up indenting entire files,
// resulting in wasted horizontal space.
#define TCNN_NAMESPACE_BEGIN namespace tcnn {
#define TCNN_NAMESPACE_END }
#include <tiny-cuda-nn/cpp_api.h>
#include <fmt/core.h>
#include <array>
#include <iostream>
#include <sstream>
#include <stdexcept>
#include <string>
#include <cuda_fp16.h>
...
and there is no fmt/core.h around it:
But I found it here:
C:\Users\user1\source\repos\Zielon\INSTA\instant-ngp\dependencies\tiny-cuda-nn\dependencies\fmt\include\fmt
Please, can you check again if all versions and commits are good since instant-ngp compiles fine for me, but not INSTA?
Sometimes the bottom row of pixels in the generated image are black.
Line 46 in 9e31e12
I think this line should be size_t i = x + y * res3d.x();
thanks!
Any Suggestion about How to improve it.
obama drive nf_01
duda drive obama
nf_01 drive nf_01
Do you know how to use one person's facial expressions to drive another person's model?
Hello!
May I ask if you have released the test dataset for the model ? How can I obtain the testing sequences portion of your dataset?
Hello, thank for your great work. I am currently working on building an live avatar system for Telepresence. Given the live recorded RGB video and tracked 3D mesh, we create and render the avatar in real-time. I think INSTA is very suitable for our scenario. The training time is very slow. The main problem is that the RGB video and mesh is pre-recorded. Could you give me some suggestions on how to enable real-time scenarios?
I have some a few guesses.
We first train a model using 2-3min RGB video and tracked mesh. As quality loss is acceptable, then we directly use the model for new-generated RGB video and mesh in real-time. Or we re-train the model using new-generated RGB video and mesh in real-time.
As the 2D video is transmitted as video chunk (1s). We train a single model for each video chunk and send the model to the user. But I guess the quality will be greatly reduced as the training set uses videos of 2-3min.
Thank you for your time. Any suggestions and comment is truly appreciable.
Wonderful work and nice dataset!
But I find the neck in the test dataset is different from the train dataset which will lead a bad metric results as the model is not trained with the larger neck.
So could please tell me how to get the same neck in train dataset or release the test dataset which has the same neck with the train dataset.
Thanks a lot!
Hi, zielon, many thanks for your great work! It's pretty awesome!
May I inquire if you could open-source the girl's datasets utilized in your paper? Cause I noticed that all datasets you released are men. I think only men may bring some potential sexism issues.
Looking forward to your reply! Many Thanks!
Best,
Xiangyue
Really amazing work! I want to try some image-to-image translation on the final renderings to improve the final quality and add the background back in. To do this, I want to create a paired dataset of real video frames and the corresponding INSTA renderings. But, when I try to render a scene using:
./build/rta --config insta.json --scene data/obama/transforms_train.json --snapshot data/obama/snapshot.msgpack --no-gui --height 512 --width 512
and
"render_novel_trajectory": false,
"render_from_snapshot": true (I've tried false here as well)
Only the test set is rendered. Is it possible to render the full dataset, including the trainset?
hello,
I am trying to train an avatar using INSTA. I disable the gui option using --no-gui since there is no gui on my machine. I wonder how to set the training steps and visualize the results?
Thanks for your solution for headless trainging, it works well.
In the Supplemental Document->B. Applications, the expression transfer results are attractive. Is it possible to do expression transfer with the existing code?
Hello, thanks for your awesome work.
I've got a question about the testing.
I've run the testing using the provided snapshot. Looks like something was rendered (100% (350/350)), but I don't see any rendered images in the /debug folder. Also, I'm getting "Can't set params because CPU buffer has the wrong size" error, although I am currently using A100 gpus. (Or is this just a warning?)
Any suggestions or solutions please?
Thank you in advance.
When creating my own dataset in the simplification.py file , I am getting an error:
ValueError: shape mismatch: value array of shape (4065,3) could not be broadcast to indexing result of shape (5023,3)
This occurs at This Line
Any ideas on how to fix it?
when I use ./build/rta Loading from a checkpoint, and I want to keep training ,it seems can not training always. or I need click the "start training", but it only can train few steps, then stop training, how can I fix this?
And another question is, when I use this GUI export mesh obj file , the result is very bad , do you have any suggestion?
Hi, thanks for sharing the excellent work!
I want to reenact the provided avatar with new expressions. So I need to use Metrical Photometric Tracker to track FLAME parameters of expression a
from a video of another avatar and generate the mesh with the combination of canonical shape parameters and expression parameters of expression a
.
However, the strange thing is that the number of faces(8090) of canonical mesh provided is not equal to that (9766) of output mesh from Metrical Photometric Tracker. I also found that the number of faces of output mesh from DECA is also 9766. Do you have any ideas on this problem?
Besides, could you please provide the FLAME parameters of each frame and canonical shape parameter in your dataset, which will make the reenactment easier (if so, no FLAME tracker is needed)?
Hi, zielon, many thanks for your reply! I fully understand that we need everyone to agree on releasing their data.
In that case, could you please provide the processed data of "yufeng" in IMavatar? I noticed that you also trained this model in your paper. But it's datasets are not aligned with your format.
Looking forward to your reply! Many Thanks!
Best,
Xiangyue
Which parameters can I change to avoid this? Any suggested parameter values?
16:35:16 INFO iteration=858 loss=0.004551
16:35:16 INFO iteration=859 loss=0.004551
16:35:16 INFO iteration=860 loss=0.004551
16:35:16 INFO iteration=861 loss=0.004551
16:35:16 INFO iteration=862 loss=0.004551
16:35:16 INFO iteration=863 loss=0.004551
16:35:16 INFO iteration=864 loss=0.004551
16:35:16 INFO iteration=865 loss=nan
16:35:16 INFO iteration=866 loss=nan
16:35:16 INFO iteration=867 loss=nan
16:35:16 INFO iteration=868 loss=nan
16:35:16 INFO iteration=869 loss=nan
16:35:16 INFO iteration=870 loss=nan
16:35:16 INFO iteration=871 loss=nan
16:35:16 INFO iteration=872 loss=nan
I have finished the training model , but I want to training more steps (for example max_steps=45), when I use --config insta.json --scene data/obama/transforms_test.json --snapshot data/obama/snapshot.msgpack to training , and I also reset the max_steps(max_steps=100) more than before, but it seems can not training ,the iteration always the same number:
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
08:37:50 INFO iteration=46 loss=0.0202575
....
and can't stop.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.