skhu101 / gauhuman Goto Github PK

Hi,
I met a problem when "bash train_zju_mocap_refine.sh". The combination is gcc12.3+cuda11.8+ninja1.10.2.4+pytorch2.0.0.

The error information is as follows：
Traceback (most recent call last):
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 1893, in _run_ninja_build
subprocess.run(
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "train.py", line 16, in
from gaussian_renderer import render, network_gui
File "C:\Users\Mr.Lin\Desktop\PROJECT\gauhuman\gaussian_renderer_init_.py", line 15, in
from scene.gaussian_model import GaussianModel
File "C:\Users\Mr.Lin\Desktop\PROJECT\gauhuman\scene_init_.py", line 16, in
from scene.dataset_readers import sceneLoadTypeCallbacks
File "C:\Users\Mr.Lin\Desktop\PROJECT\gauhuman\scene\dataset_readers.py", line 28, in
from scene.gaussian_model import BasicPointCloud
File "C:\Users\Mr.Lin\Desktop\PROJECT\gauhuman\scene\gaussian_model.py", line 23, in
from knn_cuda import KNN
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\knn_cuda_init_.py", line 38, in
knn = load_cpp_ext("knn")
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\knn_cuda_init.py", line 26, in load_cpp_ext
ext = load(
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 1284, in load
return _jit_compile(
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 1509, in _jit_compile
_write_ninja_file_and_build_library(
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 1624, in _write_ninja_file_and_build_library
_run_ninja_build(
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 1909, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'knn': [1/1] "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64/link.exe" knn.o knn.cuda.o /nologo /DLL c10.lib c10_cuda.lib torch_cpu.lib torch_cuda.lib -INCLUDE:?warp_size@cuda@at@@yahxz torch.lib /LIBPATH:C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\lib torch_python.lib /LIBPATH:C:\Users\Mr.Lin\anaconda3\envs\gauhuman\libs "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64" cudart.lib /out:knn.pyd
FAILED: knn.pyd
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64/link.exe" knn.o knn.cuda.o /nologo /DLL c10.lib c10_cuda.lib torch_cpu.lib torch_cuda.lib -INCLUDE:?warp_size@cuda@at@@yahxz torch.lib /LIBPATH:C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\lib torch_python.lib /LIBPATH:C:\Users\Mr.Lin\anaconda3\envs\gauhuman\libs "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64" cudart.lib /out:knn.pyd
Creating library knn.lib and object knn.exp
knn.o : error LNK2019: unresolved external symbol "public: long * __cdecl at::TensorBase::data_ptr(void)const " (??$data_ptr@J@TensorBase@at@@QEBAPEAJXZ) referenced in function "public: long * __cdecl at::Tensor::data(void)const " (??$data@J@Tensor@at@@QEBAPEAJXZ)
knn.pyd : fatal error LNK1120: 1 unresolved externals
ninja: build stopped: subcommand failed.

I wanted to test the render speed (FPS) of the model, but the output of the code (~300) was inconsistent with the results in the paper (100+). I made the following changes to the dataset_reader.py(558-562), but the result is still incorrect. How can I solve it?

        pose_start = 0
        # pose_interval = 30
        pose_interval = 1
        pose_num = 17

Hi, thanks for the great work and sharing code! I tested GauHuman, it was mind blowing.

From your paper, i see that equation transfroms Gaussians from camera coordinate to ray coordinate transformation in Section 3.1.

But i have not heard about ray-coordinate, i cannot fully understand what is the ray coordinate, why we need that process. (May be the ray coordinate is 2D image space?)

Would you explain about the ray coordinate and where is the code lines about transformation process?

Hello!!
Have you test your framework on dataset People_SnapShot?

Hello,

Thanks for your excellent work. I encountered some issues when using my customized dataset, specifically related to the SMPL estimations. I compared SMPL vertices visualizations as shown in the figure above. They are for monocap/lan_images620_1300/images/00/0000.jpg and from various sources:

1. Original MonoCap dataset
2. Multiple-view estimations from EasyMocap (used by MonoCap for SMPL parameter estimation)
3. Monocular estimations from EasyMocap
4. Our customized monocular SMPL estimations

When processed through GauHuman, only 1 and 2 yield correct results, while 3 and 4 consistently produce incorrect outcomes (predicted images are all black). One observed issue is the generation of erroneous 2D bound masks from 3 and 4. My speculation is that this is rooted in the differences in SMPL estimations. As shown in the figure, 1 and 2 are closely aligned, as are 3 and 4. However, the differences between these two groups are significant.

Given these observations, I’m seeking your insights –

• Could the discrepancies between multiple-view and monocular estimations, perhaps in their coordinate systems, be causing these issues?
• I’ve also noticed GauHuman computes SMPL vertices by (1) obtaining the original vertices $V$ from the SMPL models, and (2) $V=VR^T+T$, instead of directly reading the dataset’s vertices. Interestingly, they seem to be identical. Could you explain the rationale?

I am also looking forward to any additional insights that could assist in generating accurate results from monocular SMPL estimations. Thank you for your time and assistance.

Hello, thanks for your excellent work!
I met a problem when "pip install -r requirement.txt". I was stuck on installing pickle5. I also tried conda install -c conda-forge pickle5 but it doesn't work. I wnat to know if anyone else meet the same problem.

The error information is as follows：
Building wheels for collected packages: pickle5
Building wheel for pickle5 (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [38 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-38
creating build\lib.win-amd64-cpython-38\pickle5
copying pickle5\pickle.py -> build\lib.win-amd64-cpython-38\pickle5
copying pickle5\pickletools.py -> build\lib.win-amd64-cpython-38\pickle5
copying pickle5_init_.py -> build\lib.win-amd64-cpython-38\pickle5
creating build\lib.win-amd64-cpython-38\pickle5\test
copying pickle5\test\pickletester.py -> build\lib.win-amd64-cpython-38\pickle5\test
copying pickle5\test\test_pickle.py -> build\lib.win-amd64-cpython-38\pickle5\test
copying pickle5\test\test_picklebuffer.py -> build\lib.win-amd64-cpython-38\pickle5\test
copying pickle5\test_init_.py -> build\lib.win-amd64-cpython-38\pickle5\test
running build_ext
building 'pickle5._pickle' extension
creating build\temp.win-amd64-cpython-38
creating build\temp.win-amd64-cpython-38\Release
creating build\temp.win-amd64-cpython-38\Release\pickle5
"D:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -ID:\anaconda3\envs\gauhuman\include -ID:\anaconda3\envs\gauhuman\Include "-ID:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\include" "-ID:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\ATLMFC\include" "-ID:\Visual studio\2022\Community\VC\Auxiliary\VS\include" "-ID:\Windows Kits\10\include\10.0.22621.0\ucrt" "-ID:\Windows Kits\10\include\10.0.22621.0\um" "-ID:\Windows Kits\10\include\10.0.22621.0\shared" "-ID:\Windows Kits\10\include\10.0.22621.0\winrt" "-ID:\Windows Kits\10\include\10.0.22621.0\cppwinrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" /Tcpickle5/_pickle.c /Fobuild\temp.win-amd64-cpython-38\Release\pickle5/_pickle.obj
_pickle.c
"D:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -ID:\anaconda3\envs\gauhuman\include -ID:\anaconda3\envs\gauhuman\Include "-ID:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\include" "-ID:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\ATLMFC\include" "-ID:\Visual studio\2022\Community\VC\Auxiliary\VS\include" "-ID:\Windows Kits\10\include\10.0.22621.0\ucrt" "-ID:\Windows Kits\10\include\10.0.22621.0\um" "-ID:\Windows Kits\10\include\10.0.22621.0\shared" "-ID:\Windows Kits\10\include\10.0.22621.0\winrt" "-ID:\Windows Kits\10\include\10.0.22621.0\cppwinrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" /Tcpickle5/picklebufobject.c /Fobuild\temp.win-amd64-cpython-38\Release\pickle5/picklebufobject.obj
picklebufobject.c
pickle5/picklebufobject.c(19): warning C4273: “PyPickleBuffer_FromObject”: dll 链接不一致
D:\anaconda3\envs\gauhuman\include\picklebufobject.h(18): note: 参见“PyPickleBuffer_FromObject”的前一个定义
pickle5/picklebufobject.c(38): warning C4273: “PyPickleBuffer_GetBuffer”: dll 链接不一致
D:\anaconda3\envs\gauhuman\include\picklebufobject.h(22): note: 参见“PyPickleBuffer_GetBuffer”的前一个定义
pickle5/picklebufobject.c(57): warning C4273: “PyPickleBuffer_Release”: dll 链接不一致
D:\anaconda3\envs\gauhuman\include\picklebufobject.h(24): note: 参见“PyPickleBuffer_Release”的前一个定义
pickle5/picklebufobject.c(208): warning C4273: “PyPickleBuffer_Type”: dll 链接不一致
D:\anaconda3\envs\gauhuman\include\picklebufobject.h(13): note: 参见“PyPickleBuffer_Type”的前一个定义
"D:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\bin\HostX86\x64\link.exe" /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:D:\anaconda3\envs\gauhuman\libs /LIBPATH:D:\anaconda3\envs\gauhuman /LIBPATH:D:\anaconda3\envs\gauhuman\PCbuild\amd64 "/LIBPATH:D:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\ATLMFC\lib\x64" "/LIBPATH:D:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:D:\Windows Kits\10\lib\10.0.22621.0\ucrt\x64" "/LIBPATH:D:\Windows Kits\10\lib\10.0.22621.0\um\x64" /EXPORT:PyInit__pickle build\temp.win-amd64-cpython-38\Release\pickle5/_pickle.obj build\temp.win-amd64-cpython-38\Release\pickle5/picklebufobject.obj /OUT:build\lib.win-amd64-cpython-38\pickle5_pickle.cp38-win_amd64.pyd /IMPLIB:build\temp.win-amd64-cpython-38\Release\pickle5_pickle.cp38-win_amd64.lib
正在创建库 build\temp.win-amd64-cpython-38\Release\pickle5_pickle.cp38-win_amd64.lib 和对象 build\temp.win-amd64-cpython-38\Release\pickle5_pickle.cp38-win_amd64.exp
正在生成代码
已完成代码的生成
picklebufobject.obj : error LNK2005: PyPickleBuffer_GetBuffer 已经在 python38.lib(python38.dll) 中定义
build\lib.win-amd64-cpython-38\pickle5_pickle.cp38-win_amd64.pyd : fatal error LNK1169: 找到一个或多个多重定义的符号
error: command 'D:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\bin\HostX86\x64\link.exe' failed with exit code 1169
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pickle5
Running setup.py clean for pickle5
Failed to build pickle5
ERROR: Could not build wheels for pickle5, which is required to install pyproject.toml-based projects

Hi, I run the training code successfully, but after 30 iterations' training it crashes. The error occurs in the loss.backward(), and it raise an error "RuntimeError: merge_sort: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered". I have checked the losses, and it seems normal. Can you help me? Thanks a lot.

Thanks a lot for sharing this fantastic work!

Question: I see that you have added the diff-gaussian-rasterization as a submodule (https://github.com/skhu101/GauHuman/tree/main/submodules/diff-gaussian-rasterization) with the repo and not as a git submodule. I wonder if GauHuman uses a modified Gsplat rasterizer developed on top of the original repo https://github.com/graphdeco-inria/diff-gaussian-rasterization.

I was inspired by your work in the GauHuman paper where you retrained InstantAvatar on the ZJU-Mocap and MonoCap datasets. I am interested in replicating the results presented.
Could you please share the code used for this specific retraining? It would greatly assist in my research.

# get tar_pts, smpl space source pose
    A, R, Th, joints = get_transform_params_torch(self.SMPL_NEUTRAL, params, rot_mats=rot_mats)

    self.s_A = A
    A = torch.matmul(bweights, self.s_A.reshape(bs, joints_num, -1))
    A = torch.reshape(A, (bs, -1, 4, 4))
    can_pts = torch.matmul(A[..., :3, :3], query_pts[..., None]).squeeze(-1)
    smpl_src_pts = can_pts + A[..., :3, 3]

    transforms = torch.matmul(A[..., :3, :3], transforms)

    if return_transl:
        translation = torch.matmul(A[..., :3, :3], translation[..., None]).squeeze(-1) + A[..., :3, 3]

# transform points from the smpl space to the world space
  R_inv = torch.inverse(R)
  world_src_pts = torch.matmul(smpl_src_pts, R_inv) + Th
  
  transforms = torch.matmul(R, transforms) #< --- THIS LINE

  if return_transl: 
      translation = torch.matmul(translation, R_inv).squeeze(-1) + Th

@skhu101 can you tell me why is transforms post multiplied with R even though we are pre multiplying smpl_src_pts by R_inv ?

Hi,Id like to calculate signed distances for world_src_pts in gaussian_model.py,so I can know whether this point is inside body or not.Could you tell me how to achieve this roughly?

hello! I've successfully reproduced your project, but I'm not sure about one detail. Do you convert the SMPL vertices in the target space to the canonical space and then perform Gaussian optimization?

Hi,I want to project the world_src_pts onto 2D image plane,and I use the projection() function from your previous work SHERF,and the projected 2d points seems like wrong,could you help me solve this?Following is my code:

src_uv = projection(world_src_pts.reshape(bs, -1, 3), camera_R, camera_T, camera_K) # [bs, N, 6890, 3]
src_uv = src_uv.view(-1, *src_uv.shape[2:])

Here camera_K is the camera intrinsic'),camera_R is the ['R'] in smpl_param,camera_T is the ['Th'] in smpl_param.

Hi, seems there should be zjumocap related data.py files in data folder?

Hi @skhu101 ,

Thx for the great advance.
May I ask, the extrinsic param R (from annots.npy) seems not to be used, but smpl_param.Rh (from smpl_params/*.npy) plays the role. May u provide more info on this?

Thx & Rgds,

Hi, thanks for your excellent work!
I would like to know if this method can generate a new pose. Or how about the generalization ability under new postures?

When I run train_zju_mocap_refine.sh, an error occurs: The issue seems to arise when attempting to reshape the array, as its size does not match the expected shape of (6890, 3).

If I could receive your assistance, I would be extremely grateful.

“Traceback (most recent call last):
File "C:\Users\Razer\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 1666, in _run_ninja_build
subprocess.run(
File "C:\Users\Razer\anaconda3\envs\gauhuman\lib\subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:
......
File "C:\Users\Razer\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 1681, in _run_ninja_build
message += f": {error.output.decode()}" # type: ignore[union-attr]
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd5 in position 1179: invalid continuation byte

Anyone else having the same issue?

@skhu101 can you share for how many iterations has the final model for zju_mocap_refine data set been trained that is being used in the paper ? Is it 1200 iterations as mentioned in the train_zju_mocap_refine.sh file ?

Dear authors! Thanks for your excellent work.
I have a question about the mask loss. I am confused about the code. Is the bkgd_mask and alpha mask used to compute the mask loss? Where is the ground truth mask? And the bound_mask=1 means the person region or others?
By the way, when inference, do we neek the mask as input?

Can you tell me the command line in train_zju_mocap_refine.sh has -eval and eval_zju_mocap_refine.sh does not have the -eval grin?
I see the code without -eval, test_cam_infos = [] would be equal to null. Wouldn't there be no new perspective rendering.

Traceback (most recent call last):
File "train.py", line 16, in
from gaussian_renderer import render, network_gui
File "C:\Users\Mr.Lin\Desktop\PROJECT\gauhuman\gaussian_renderer_init_.py", line 15, in
from scene.gaussian_model import GaussianModel
File "C:\Users\Mr.Lin\Desktop\PROJECT\gauhuman\scene_init_.py", line 16, in
from scene.dataset_readers import sceneLoadTypeCallbacks
File "C:\Users\Mr.Lin\Desktop\PROJECT\gauhuman\scene\dataset_readers.py", line 28, in
from scene.gaussian_model import BasicPointCloud
File "C:\Users\Mr.Lin\Desktop\PROJECT\gauhuman\scene\gaussian_model.py", line 23, in
from knn_cuda import KNN
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\knn_cuda_init_.py", line 38, in
knn = load_cpp_ext("knn")
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\knn_cuda_init.py", line 26, in load_cpp_ext
ext = load(
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 1310, in load
return _jit_compile(
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 1738, in _jit_compile
return _import_module_from_library(name, build_directory, is_python_module)
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 2138, in _import_module_from_library
module = importlib.util.module_from_spec(spec)
ImportError: DLL load failed while importing knn: 找不到指定的模块。

Why the code can be run directly in vscode, but a RuntimeError occurs when debugging:

发生异常: RuntimeError
Error building extension 'knn': [1/3] /usr/local/cuda-11.8:/usr/local/cuda-11.8/bin/nvcc -DTORCH_EXTENSION_NAME=knn -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/TH -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda-11.8:/usr/local/cuda-11.8/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_89,code=compute_89 -gencode=arch=compute_89,code=sm_89 --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -std=c++17 -c /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cu -o knn.cuda.o
FAILED: knn.cuda.o
/usr/local/cuda-11.8:/usr/local/cuda-11.8/bin/nvcc -DTORCH_EXTENSION_NAME=knn -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/TH -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda-11.8:/usr/local/cuda-11.8/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_89,code=compute_89 -gencode=arch=compute_89,code=sm_89 --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -std=c++17 -c /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cu -o knn.cuda.o
/bin/sh: 1: /usr/local/cuda-11.8:/usr/local/cuda-11.8/bin/nvcc: not found
[2/3] c++ -MMD -MF knn.o.d -DTORCH_EXTENSION_NAME=knn -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/TH -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda-11.8:/usr/local/cuda-11.8/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O2 -c /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cpp -o knn.o
FAILED: knn.o
c++ -MMD -MF knn.o.d -DTORCH_EXTENSION_NAME=knn -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/TH -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda-11.8:/usr/local/cuda-11.8/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O2 -c /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cpp -o knn.o
In file included from /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cpp:3:
/home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/ATen/cuda/CUDAContext.h:5:10: fatal error: cuda_runtime_api.h: No such file or directory
5 | #include <cuda_runtime_api.h>
| ^~~~~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

File "/home/qinzhen/GauHuman/scene/gaussian_model.py", line 23, in
from knn_cuda import KNN
File "/home/qinzhen/GauHuman/scene/dataset_readers.py", line 28, in
from scene.gaussian_model import BasicPointCloud
File "/home/qinzhen/GauHuman/scene/init.py", line 16, in
from scene.dataset_readers import sceneLoadTypeCallbacks
File "/home/qinzhen/GauHuman/gaussian_renderer/init.py", line 15, in
from scene.gaussian_model import GaussianModel
File "/home/qinzhen/GauHuman/train.py", line 16, in
from gaussian_renderer import render, network_gui
RuntimeError: Error building extension 'knn': [1/3] /usr/local/cuda-11.8:/usr/local/cuda-11.8/bin/nvcc -DTORCH_EXTENSION_NAME=knn -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/TH -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda-11.8:/usr/local/cuda-11.8/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_89,code=compute_89 -gencode=arch=compute_89,code=sm_89 --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -std=c++17 -c /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cu -o knn.cuda.o
FAILED: knn.cuda.o
/usr/local/cuda-11.8:/usr/local/cuda-11.8/bin/nvcc -DTORCH_EXTENSION_NAME=knn -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/TH -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda-11.8:/usr/local/cuda-11.8/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_89,code=compute_89 -gencode=arch=compute_89,code=sm_89 --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -std=c++17 -c /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cu -o knn.cuda.o
/bin/sh: 1: /usr/local/cuda-11.8:/usr/local/cuda-11.8/bin/nvcc: not found
[2/3] c++ -MMD -MF knn.o.d -DTORCH_EXTENSION_NAME=knn -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/TH -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda-11.8:/usr/local/cuda-11.8/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O2 -c /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cpp -o knn.o
FAILED: knn.o
c++ -MMD -MF knn.o.d -DTORCH_EXTENSION_NAME=knn -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/TH -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda-11.8:/usr/local/cuda-11.8/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O2 -c /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cpp -o knn.o
In file included from /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cpp:3:
/home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/ATen/cuda/CUDAContext.h:5:10: fatal error: cuda_runtime_api.h: No such file or directory
5 | #include <cuda_runtime_api.h>
| ^~~~~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.

Hi,I wander how to change the size of output image in your network,should I modify the gaussian rasterizer in CUDA?

Hi,What an amazing job!
I have some questions. Do you have any plans to release the code for making the dataset? And more,for ZJU_mocap DATASETS?
And more, I observe the value of SEQUENCES in the bash : train_zju_mocap_refine is mismatched. Do I need to make some modifications?

How do I make my own video dataset?

Hello there,
Could you please tell me what the problem is?
I really follow your script, but When I run train_zju.sh, I got something wrong:

/data/xhd/envs/gaussian/lib/python3.7/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing weights=VGG16_Weights.IMAGENET1K_V1. You can also use weights=VGG16_Weights.DEFAULT to get the most up-to-date weights.
warnings.warn(msg)
Loading model from: /data/xhd/envs/gaussian/lib/python3.7/site-packages/lpips/weights/v0.1/vgg.pth
Optimizing
Output folder: ./output/zju_mocap_refine/386_100_pose_correction_lbs_offset_split_clone_merge_prune [05/04 15:41:47]
Tensorboard not available: not logging progress [05/04 15:41:47]
Found annots.json file, assuming ZJU_MoCap_refine data set! [05/04 15:41:47]
Reading Training Transforms [05/04 15:41:47]
Reading Test Transforms [05/04 15:41:54]
Loading Training Cameras [05/04 15:42:20]
Loading Test Cameras [05/04 15:42:24]
Number of points at initialisation : 6890 [05/04 15:42:27]
Training progress: 0%| | 0/1200 [00:00<?, ?it/s]Traceback (most recent call last):
File "train.py", line 301, in
training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from)
File "train.py", line 104, in training
render_pkg = render(viewpoint_cam, gaussians, pipe, background)
File "/data/xhd/3D/GauHuman/gaussian_renderer/init.py", line 119, in render
cov3D_precomp = cov3D_precomp)
ValueError: not enough values to unpack (expected 4, got 2)
Training progress: 0%|

Excellent work!
But if I want to train gauhuman on the custom dataset, how can I get or train the SMPL model of the custom dataset?

Hi,I want to render all the resulting images with a fixed camera,
Here is my code, i just change the parameters about the camera

# lights is the fixed camera
   if lights is not None:
       # Set up rasterization configuration
       tanfovx = math.tan(lights[0].FoVx * 0.5)
       tanfovy = math.tan(lights[0].FoVy * 0.5)

       raster_settings = GaussianRasterizationSettings(
           image_height=int(lights[0].height),# change
           image_width=int(lights[0].width),# change
           tanfovx=tanfovx,   # change
           tanfovy=tanfovy,
           bg=bg_color,
           scale_modifier=scaling_modifier,
           viewmatrix=lights[0].viewmatrix,# change
           projmatrix=lights[0].projmatrix,# change
           sh_degree=pc.active_sh_degree,
           campos=lights[0].position,# change
           prefiltered=False,
           debug=pipe.debug
       )

   else:
       # Set up rasterization configuration
       tanfovx = math.tan(viewpoint_camera.FoVx * 0.5)
       tanfovy = math.tan(viewpoint_camera.FoVy * 0.5)

       raster_settings = GaussianRasterizationSettings(
           image_height=int(viewpoint_camera.image_height),
           image_width=int(viewpoint_camera.image_width),
           tanfovx=tanfovx,
           tanfovy=tanfovy,
           bg=bg_color,
           scale_modifier=scaling_modifier,
           viewmatrix=viewpoint_camera.world_view_transform,
           projmatrix=viewpoint_camera.full_proj_transform,
           sh_degree=pc.active_sh_degree,
           campos=viewpoint_camera.camera_center,
           prefiltered=False,
           debug=pipe.debug
       )

   rasterizer = GaussianRasterizer(raster_settings=raster_settings)

and i got some missing in results

could you tell me how to deal with it?

After set the environment,I run this script "bash run_zju_mocap_refine.sh",and encounter:
Traceback (most recent call last):
File "train.py", line 16, in
from gaussian_renderer import render, network_gui
File "/home/yejr/Digital_Avater/GauHuman-main/gaussian_renderer/init.py", line 15, in
from scene.gaussian_model import GaussianModel
File "/home/yejr/Digital_Avater/GauHuman-main/scene/init.py", line 16, in
from scene.dataset_readers import sceneLoadTypeCallbacks
File "/home/yejr/Digital_Avater/GauHuman-main/scene/dataset_readers.py", line 33, in
from data.dna_rendering.dna_rendering_sample_code.SMCReader import SMCReader
ModuleNotFoundError: No module named 'data.dna_rendering'
How to deal with it?

Hi!
I encountered an issue while running the training code using the provided train_zju_mocap_refine.sh script on a single Nvidia RTX 4090 GPU. It appears that the rasterizer is demanding 72 GB GPU memory.

I'm reaching out to inquire if any specific parameters within the script or configuration can be adjusted to reduce the GPU memory requirements to a more manageable level.

Thank you for your assistance.

Thanks for the amazing work!

There is some bugs that i can not train on monocap.

But it is ok when i use vlad_images1011,

I am so confused.

Hi,I wonder how to use the interactive viewers from original 3DGS in gauhuman?

Hi,

I download the 'my_377' of ZJU-MoCap dataset from instant-nvr. But the folder only contains annots_new.npy instead of annots.npy in your code.

If I change the file name in the code and load annots_new.npy, annots only contains 'cams' while 'ims' is empty.

Thanks a lot for your great work!

I visualize the initialization of gaussians (big pose xyz). I save these point clouds and view them in meshlab.

The result is as this, all points are spread out. I think it should be dense point clouds like the canonical space gaussian in Fig 2 of the paper.

Great work.
Have you ever tested KL split/clone performance in MVS scenarios, like static scenes with multi-camera?
When I try it, I see that the number of 3dgs has increased, not decreased, and the running time has not changed significantly. Is there something wrong?