nvidiagameworks / kaolin-wisp Goto Github PK

NVIDIA Kaolin Wisp is a PyTorch library powered by NVIDIA Kaolin Core to work with neural fields (including NeRFs, NGLOD, instant-ngp and VQAD).

License: Other

Python 92.84% Groovy 1.53% Dockerfile 0.06% C++ 1.61% Cuda 3.94% Shell 0.02%

3d-deep-learning nerf pytorch sdf neuralfields

kaolin-wisp's People

Contributors

Stargazers

Watchers

Forkers

tontontremblay gatsby23 ssground jasonlsc ray8828 caenorst orperel zenotech mahsa13473 gvc0461082002 nirvanalan ashishd joeylitalien techthiyanes 0xgear0 akankushjnvku tovacinni omwtoswe mbrukman tmats doubleeint poomink pkhemapet lyrl xiongdao1 0xlian0 shidaoe sulwon-0516 jaedukseo cuulee cnheider jamesthesnake marcelsan idaho777 protericura bruinxiong cubantonystark chomolungma satoshirobatofujimoto simrit1 tuna1210 ajunlonglive v-wewei salarios77 yasasa kevinhangoat shota3506 jafffy peterzs jryanshue yasutomo57jp chrischoy nkyriazis eric-heiden 3a1b2c3 stnamjef pooncs catherineytw soumik12345 herculesyuan phoenixdigitalfx robo-warrior2 mwang625 mvsc1 gg-big-org pterameta cemberk derrick-xwp ehedlin weichengtseng ogata-lab hmthanh ricklentz baquer anabur920 dinhanhtruong wizyke gavinguan95 bilibilicancer jakobtroidl flufy3d docsavage svsambandam jeremysxy hxl3s yair-mantis zenos4mbu shumash galhar fred62879 zzboya abecadel domaradzkimaciej friedrich-m jyh2005xx avivsham alvaro-budria jaeheungs x-oldgentleman imclab

kaolin-wisp's Issues

module 'kaolin._C.ops.spc' has no attribute 'points_to_morton_cuda'

Hello! I'm trying to get kaolin-wisp working on a GCP Compute Engine VM and everything installs without error, but when I run
python3 app/main.py --config configs/ngp_nerf.yaml --multiview-dataset-format standard --mip 0 --dataset-path ../instant-ngp/data/nerf/fox

I get an error:
AttributeError: module 'kaolin._C.ops.spc' has no attribute 'points_to_morton_cuda'

I ran dir() on kaolin._C.ops.spc and I did not see points_to_morton_cuda:
['Conv3d_backward', 'Conv3d_forward', 'ConvTranspose3d_backward', 'ConvTranspose3d_forward', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'generate_points_cuda', 'points_to_octree', 'scan_octrees_cuda', 'to_dense_backward', 'to_dense_forward']

Any idea what might be going on?

about the param 'at' of Camera.from_args()

first, thanks for your excellent work.
But, could you tell me how to compute the param 'at' for Camera.from_args() from other datasets like 'DTU'?

cannot import name 'Camera' from 'kaolin.render.camera'

Hi, thanks for this wonderful project code release!

I followed the instructions and installed kaolin-wisp, but when I import wisp I get:

    >>> import wisp
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/yifu/workspace/kaolin-wisp/wisp/__init__.py", line 10, in <module>
        import wisp.config_parser
      File "/home/yifu/workspace/kaolin-wisp/wisp/config_parser.py", line 15, in <module>
        from wisp.datasets import *
      File "/home/yifu/workspace/kaolin-wisp/wisp/datasets/__init__.py", line 10, in <module>
        from .multiview_dataset import MultiviewDataset
      File "/home/yifu/workspace/kaolin-wisp/wisp/datasets/multiview_dataset.py", line 13, in <module>
        from wisp.datasets.formats import load_nerf_standard_data, load_rtmv_data
      File "/home/yifu/workspace/kaolin-wisp/wisp/datasets/formats/__init__.py", line 9, in <module>
        from .rtmv import *
      File "/home/yifu/workspace/kaolin-wisp/wisp/datasets/formats/rtmv.py", line 18, in <module>
        from kaolin.render.camera import Camera, blender_coords
    ImportError: cannot import name 'Camera' from 'kaolin.render.camera' (/home/yifu/workspace/kaolin/kaolin/render/camera.py)

Importing kaolin (0.11.0) itself is working for me. I am using Python 3.8.13, PyTorch 1.12 with cuda 11.6.

Thanks in advance!

error C2398: element '2': conversion from 'unsigned __int64' to '_Ty' requires a narrowing conversion

hello!Thanks for the great project!
I'm having a problem configuring the project environment.
------------------------------------------------------------------------------>
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include\stdint.h(21): note:See declaration of "int64_t"
wisp/csrc\ops\hashgrid_interpolate.cpp(49): error C2398: element '2': conversion from 'unsigned __int64' to '_Ty' requires a narrowing conversion
with
[
_Ty=int64_t
]
error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl .exe' failed with exit status 2
above!
////////////////////////////////////////////////////////////////////////////////////
above!
my environment is：
windows11
python=3.8
pytorch=1.11.0
cuda=11.3
gpu=NVIDIA GeForce GTX 1650
I followed the README.md instructions exactly and installed steps 1-6 successfully. Error in step 7.
can you tell me what to do？

spc_render.diff returns illegal memory access

Running spc_render.diff returns

RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

load_nerf_standard_data() assertion error for standard nerf dataset

I tried to run the fox example under the Training NGP for forward facing scenes section and I'm getting the following error:

2022-08-18 15:41:18,934|    INFO| EPOCH 47/50 | total loss: 2.888E-02 | rgb loss: 2.888E-02
2022-08-18 15:41:31,912|    INFO| EPOCH 48/50 | total loss: 2.786E-02 | rgb loss: 2.786E-02
2022-08-18 15:41:44,976|    INFO| EPOCH 49/50 | total loss: 2.824E-02 | rgb loss: 2.824E-02
2022-08-18 15:41:58,006|    INFO| EPOCH 50/50 | total loss: 2.846E-02 | rgb loss: 2.846E-02
2022-08-18 15:42:03,052|    INFO| Saving model checkpoint to: _results/logs/runs/test-ngp-nerf/20220818-153021/model.pth
2022-08-18 15:42:03,516|    INFO| Beginning validation...
Traceback (most recent call last):
  File "/home/joonho/workspace/repos/kaolin-wisp/wisp/trainers/base_trainer.py", line 314, in iterate
    data = self.next_batch()
  File "/home/joonho/workspace/repos/kaolin-wisp/wisp/trainers/base_trainer.py", line 304, in next_batch
    return next(self.train_data_loader_iter)
  File "/home/joonho/miniconda3/envs/kaolin/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 530, in __next__
    data = self._next_data()
  File "/home/joonho/miniconda3/envs/kaolin/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 569, in _next_data
    index = self._next_index()  # may raise StopIteration
  File "/home/joonho/miniconda3/envs/kaolin/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "app/main.py", line 35, in <module>
    trainer.train()
  File "/home/joonho/workspace/repos/kaolin-wisp/wisp/trainers/base_trainer.py", line 449, in train
    self.iterate()
  File "/home/joonho/workspace/repos/kaolin-wisp/wisp/trainers/base_trainer.py", line 317, in iterate
    self.end_epoch()
  File "/home/joonho/workspace/repos/kaolin-wisp/wisp/trainers/base_trainer.py", line 287, in end_epoch
    self.validate(self.epoch)
  File "/home/joonho/workspace/repos/kaolin-wisp/wisp/trainers/multiview_trainer.py", line 162, in validate
    data = self.dataset.get_images(split="val", mip=2)
  File "/home/joonho/workspace/repos/kaolin-wisp/wisp/datasets/multiview_dataset.py", line 92, in get_images
    data = load_nerf_standard_data(self.root, split,
  File "/home/joonho/workspace/repos/kaolin-wisp/wisp/datasets/formats/nerf_standard.py", line 127, in load_nerf_standard_data
    assert False and f"Split type ['{split}'] unsupported in the dataset provided"
AssertionError

Steps to reproduce:

Clone instant-ngp repo to obtain fox dataset
run python3 app/main.py --config configs/ngp_nerf.yaml --multiview-dataset-format standard --mip 0 --dataset-path /path/to/fox

Possible Cause:
load_nerf_standard_data has an if-statement block that checks len(transforms) to determine what keys to insert into transform_dict.

kaolin-wisp/wisp/datasets/formats/nerf_standard.py

Lines 103 to 127 in 729608a

 transforms = sorted(glob.glob(os.path.join(root, "*.json"))) 

 transform_dict = {} 

 train_only = False 

 if mip is None: 

 mip = 0 

 if len(transforms) == 1: 

 transform_dict['train'] = transforms[0] 

 train_only = True 

 elif len(transforms) == 3: 

 fnames = [os.path.basename(transform) for transform in transforms] 

 # Create dictionary of split to file path, probably there is simpler way of doing this 

 for _split in ['test', 'train', 'val']: 

 for i, fname in enumerate(fnames): 

 if _split in fname: 

 transform_dict[_split] = transforms[i] 

 else: 

 assert False and "Unsupported number of splits, there should be ['test', 'train', 'val']" 

 if split not in transform_dict: 

 assert False and f"Split type ['{split}'] unsupported in the dataset provided"

The fox standard nerf dataset provided by instant-ngp has only 1 transform.json so the transforms list has one value (transform.json). This causes the first if-statement to be true and inserts transforms_dict['train'] = transforms[0]. The next if-statement on line 126 checks if transforms_dict has the split key, but because this is all being called under the self.validate(self.epoch) function (from the stacktrace), split = val and the assertion fails since we hard-coded to insert the key train.

Installation with CUDA 11.2

Is it possible to install wisp for CUDA 11.2 ?

Config parser does not support FunnyNeuralField when running the latent nerf example

Great work on this library!!

One issue I encountered is that the config parser does not support the FunnyNeuralField class in the latent nerf example. It raises a value error and the example can't run. Simply adding FunnyNeural Field to the parser won't work since it has no init function and the base_nef init function won't initialize it properly.

kaolin-wisp/wisp/config_parser.py

Lines 586 to 614 in d118afd

 def load_nef(args, grid): 

 if args.nef_type == 'NeuralRadianceField': 

 return NeuralRadianceField( 

 grid=grid, 

 pos_embedder=args.embedder_type, 

 view_embedder=args.embedder_type, 

 pos_multires=args.pos_multires, 

 position_input=args.position_input, 

 view_multires=args.view_multires, 

 activation_type=args.activation_type, 

 layer_type=args.layer_type, 

 hidden_dim=args.hidden_dim, 

 num_layers=args.num_layers, 

 prune_density_decay=args.prune_density_decay, 

 prune_min_density=args.prune_min_density 

 ) 

 elif args.nef_type == 'NeuralSDF': 

 return NeuralSDF( 

 grid=grid, 

 pos_embedder=args.embedder_type, 

 pos_multires=args.pos_multires, 

 position_input=args.position_input, 

 activation_type=args.activation_type, 

 layer_type=args.layer_type, 

 hidden_dim=args.hidden_dim, 

 num_layers=args.num_layers 

 ) 

 else: 

 raise ValueError(f'Neural field {args.nef_type} is temporarily unsupported by the config parser.')

Is occupancy grid utilized in NeRF implementation?

Hi, in the instant-ngp paper, there is an occupancy grid updated and used in NeRF ray marching. But I think this is not implemented yet in wisp? Would you mind give more details about difficulties when implementing it and why it cannot work? Thank you so much.

Support for unbounded scenes

Hi! Is there going to be support for handling unbounded scenes via background models with re-parameterized inputs in the near future? If not, would it be possible for you to provide some tips on how to best implement that?

Support for multiple intrinsics and different image sizes

Hi I am coming from Instant-ngp, and lately, they added the capability to load individual intrinsics (by moving the global intrinsics in the json to each image). This also allows for loading different size images (or rotated images). I was wondering if there is any plans to add this capability? Thanks.

Is it possible to run APP GUI in WSL？

Hello, I tried to run Kaolin-wisp with GUI app in WSL2.

I use X11 server on my Windows11 to display the window.

But it seems not to work.

I run the example as python3 main_demo.py --dataset-path /home/XXX/data/nerf_synthetic/lego.

[i] Using GLFW (GL 2.1)
Traceback (most recent call last):
  File "/home/XXX/codes/kaolin-wisp/examples/latent_nerf/main_demo.py", line 10, in <module>
    setup_cuda_context()  # Must be called before any torch operations take place
  File "/home/XXX/codes/kaolin-wisp/wisp/cuda_guard.py", line 39, in setup_cuda_context
    import pycuda.gl.autoinit
  File "/home/XXX/codes/pycuda/pycuda/gl/autoinit.py", line 10, in <module>
    context = make_default_context(lambda dev: cudagl.make_context(dev))
  File "/home/XXX/codes/pycuda/pycuda/tools.py", line 215, in make_default_context
    return ctx_maker(dev)
  File "/home/XXX/codes/pycuda/pycuda/gl/autoinit.py", line 10, in <lambda>
    context = make_default_context(lambda dev: cudagl.make_context(dev))
pycuda._driver.LogicError: cuGLCtxCreate failed: OS call failed or operation not supported on this OS

I just know the pycuda is used for display and it works at the GPU driver level.

For example, it will copy the RenderBuffer results to the screen buffer.

So, it is impossible to display the app window from WSL2, is right?

(I also tried to run Kaolin-wisp without GUI app in WSL2, and it works well.)

My device is GeForce 4090.

Error using octree with multiscale_type='cat'

First, thank you for the great library!

Using OctreeGrid with multiscale_type='cat' gave me an error when calling interpolate at L273

kaolin-wisp/wisp/models/grids/octree_grid.py

Line 273 in 7ca5c6f

return feats.reshape(batch, num_samples, self.feature_dim)

The problem is that self.feature_dim is too small as the entries have self.feature_dim * num_feats dimensions.

I changed the code as follows, which seemed to fix it.

diff --git a/wisp/models/grids/octree_grid.py b/wisp/models/grids/octree_grid.py
index 7c6a8bc..95ec5e3 100644
--- a/wisp/models/grids/octree_grid.py
+++ b/wisp/models/grids/octree_grid.py
@@ -263,14 +263,17 @@ class OctreeGrid(BLASGrid):

             if self.multiscale_type == 'sum':
                 feats = feats.reshape(*feats.shape[:-1], num_feats, self.feature_dim)
+                out_features = self.feature_dim
                 if self.training:
                     feats = feats.sum(-2)
                 else:
                     feats = feats.sum(-2)
+            else:  # self.multiscale_type == 'cat'
+                out_features = self.feature_dim * num_feats

             timer.check("aggregate")

-            return feats.reshape(batch, num_samples, self.feature_dim)
+            return feats.reshape(batch, num_samples, out_features)

     def raymarch(self, rays, level=None, num_samples=64, raymarch_type='voxel'):
         """Mostly a wrapper over OctreeAS.raymarch. See corresponding function for more details.

GLError: glGetUniformLocation when running Interactive Training example

I'm trying to run the interactive training example:
WISP_HEADLESS=0 python3 app/main_interactive.py --config configs/nglod_nerf_interactive.yaml --dataset-path /path/to/V8 --dataset-num-workers 4 but I'm receiving the following issue:

Traceback (most recent call last):
  File "app/main_interactive.py", line 12, in <module>
    setup_cuda_context()     # Must be called before any torch operations take place
  File "/ssd003/home/joonho/workspace/repos/kaolin-wisp/app/cuda_guard.py", line 29, in setup_cuda_context
    window = app.Window(width=10, height=10, title='dummy', visible=False)
  File "/h/joonho/.conda/envs/kaolin/lib/python3.8/site-packages/glumpy/app/__init__.py", line 188, in __new__
    window = __backend__.Window(*args, **kwargs)
  File "/h/joonho/.conda/envs/kaolin/lib/python3.8/site-packages/glumpy/app/window/backends/backend_glfw_imgui.py", line 248, in __init__
    self.imguiRenderer = GlfwRenderer(self._native_window, attach_callbacks=False)
  File "/h/joonho/.conda/envs/kaolin/lib/python3.8/site-packages/imgui/integrations/glfw.py", line 13, in __init__
    super(GlfwRenderer, self).__init__()
  File "/h/joonho/.conda/envs/kaolin/lib/python3.8/site-packages/imgui/integrations/opengl.py", line 60, in __init__
    super(ProgrammablePipelineRenderer, self).__init__()
  File "/h/joonho/.conda/envs/kaolin/lib/python3.8/site-packages/imgui/integrations/base.py", line 17, in __init__
    self._create_device_objects()
  File "/h/joonho/.conda/envs/kaolin/lib/python3.8/site-packages/imgui/integrations/opengl.py", line 108, in _create_device_objects
    self._attrib_location_tex = gl.glGetUniformLocation(self._shader_handle, "Texture")
  File "/h/joonho/.conda/envs/kaolin/lib/python3.8/site-packages/OpenGL/latebind.py", line 63, in __call__
    return self.wrapperFunction( self.baseFunction, *args, **named )
  File "/h/joonho/.conda/envs/kaolin/lib/python3.8/site-packages/OpenGL/GL/VERSION/GL_2_0.py", line 431, in glGetUniformLocation
    return baseOperation( program, name )
  File "/h/joonho/.conda/envs/kaolin/lib/python3.8/site-packages/OpenGL/platform/baseplatform.py", line 416, in __call__
    return self( *args, **named )
  File "/h/joonho/.conda/envs/kaolin/lib/python3.8/site-packages/OpenGL/error.py", line 231, in glCheckError
    raise self._errorClass(
OpenGL.error.GLError: GLError(
        err = 1282,
        description = b'invalid operation',
        baseOperation = glGetUniformLocation,
        cArguments = (1, b'Texture\x00'),
        result = -1
)

Looking at the documentation for glGetUniformLocation, I see the error section say

GL_INVALID_OPERATION is generated if program is not a program object.

GL_INVALID_OPERATION is generated if program has not been successfully linked.

where GL_INVALID_OPERATION is error code 0x0502 == 1282.

I've setup my conda environment to successfully run the Training NGLOD-NeRF from multiview RGB-D data on a remote server over ssh tunnel + VNC Viewer but cant get the interactive mode working.

Loss did not drop when training VQAD from standard NeRF blender dataset

Thanks for this unique library!
I am using nerf blender data to train the VQAD model.
At first, I used the V8 data and the code worked well.
Then, I tried to use the blender dataset. However, this time, the loss didn't drop. The rendered image is just white and seems to overfit the background.

...
2022-08-22 15:20:12,092|    INFO| EPOCH 41/50 | total loss: 1.925E-01 | rgb loss: 1.925E-01
2022-08-22 15:20:22,690|    INFO| EPOCH 42/50 | total loss: 1.914E-01 | rgb loss: 1.914E-01
2022-08-22 15:20:33,307|    INFO| EPOCH 43/50 | total loss: 1.943E-01 | rgb loss: 1.943E-01
2022-08-22 15:20:43,916|    INFO| EPOCH 44/50 | total loss: 1.931E-01 | rgb loss: 1.931E-01
2022-08-22 15:20:54,543|    INFO| EPOCH 45/50 | total loss: 1.927E-01 | rgb loss: 1.927E-01
2022-08-22 15:21:05,190|    INFO| EPOCH 46/50 | total loss: 1.943E-01 | rgb loss: 1.943E-01
2022-08-22 15:21:15,847|    INFO| EPOCH 47/50 | total loss: 1.940E-01 | rgb loss: 1.940E-01
2022-08-22 15:21:26,473|    INFO| EPOCH 48/50 | total loss: 1.938E-01 | rgb loss: 1.938E-01
2022-08-22 15:21:37,133|    INFO| EPOCH 49/50 | total loss: 1.933E-01 | rgb loss: 1.933E-01
2022-08-22 15:21:47,750|    INFO| EPOCH 50/50 | total loss: 1.923E-01 | rgb loss: 1.923E-01

Only the dataset part of your vqad_nerf.yaml file is being changed.

dataset:
  dataset_type: "multiview"
  num_rays_sampled_per_img: 4096
  # multiview_dataset_format: "rtmv"
  multiview_dataset_format: "standard"
  mip: 2
  bg_color: "white"

My Environment

CUDA v11.3
ubuntu 20.04
GeForce RTX 3090

Thanks in advance!

Best way generate a video.

New to this library, can use some guidance here.

I have a dataset created using colman script. How to best train/load/save model so that I can query it to get images for different viewpoints, and then stitch it to get a video? Thanks!

Will wisp support ngp (hash grid ) with sdf ?

Hi, thanks for your great work! will wisp support that training ngp multi hash grid and joint sdf ?

support for MipNeRF360

Thanks for your awesome project. Is there any plan to support the MipNeRF360 model/dataset?

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Thanks for this amazing library!
I tried to train the VQAD from the RTMV dataset and I'm getting the following error:

$python3 app/main.py --config configs/vqad_nerf.yaml --dataset-path ../dataset/rtmv/amazon_berkely-003/00001/
...
Traceback (most recent call last):
  File "app/main.py", line 35, in <module>
    trainer.train()
  File "/home/zaima/research/kaolin-wisp/wisp/trainers/base_trainer.py", line 473, in train
    self.iterate()
  File "/home/zaima/research/kaolin-wisp/wisp/trainers/base_trainer.py", line 340, in iterate
    self.step(self.epoch, self.iteration, data)
  File "/home/zaima/research/kaolin-wisp/wisp/trainers/multiview_trainer.py", line 88, in step
    self.scaler.scale(loss).backward()
  File "/home/zaima/anaconda3/envs/wisp/lib/python3.8/site-packages/torch/_tensor.py", line 363, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/zaima/anaconda3/envs/wisp/lib/python3.8/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

My Environment

CUDA v11.3
ubuntu 20.04
GeForce RTX 3090

I've already tried to train that from V8 dataset and it finished successfully.
I checked this RTMV dataset class https://github.com/NVIDIAGameWorks/kaolin-wisp/blob/main/wisp/datasets/formats/rtmv.py , but I don't understand why I can't use RTMV datasets other than V8.
Thanks in advance!

Problems setting up wisp (with GUI) on Windows

First, thank you for the work!

I'm having some troubles setting up the library in windows 10 with the interactive GUI working.

First I just tried to install the library using the latest version of each requirement without paying too much attention at the specific version. I tried CUDA 12.0 (with Visual Studio 2022) and pytorch 1.13.1cu116, and I was able to install the wisp library but when I tried to install pycuda to use the interactive mode the problems started: apparently the git version doesn't support windows (it doesn't load the CUDA path correctly) so I installed it using pipwin install pycuda, which made the installation work but when I tried to launch some examples I got
RuntimeError: make_default_context() wasn't able to create a context on any of the 1 detected devices
same as #60, for which there is still no solution in windows (the proposed solution doesn't work)

I speculated that the problem was with the version of the libraries I installed, and I noriced that kaolin requires torch >= 1.8, <= 1.12.1, and I immagined that it was better to match the cuda version used by torch and pycuda with the one installed manually in the system, so I installed:
CUDA 11.6 (with Visual Studio 2019, that doesn't support 2022)
pytorch 1.12.1cu116
pycuda-2022.1+cuda116 (installed through pipwin)
but now trying to install the wisp package through the setup.py fails with

C:\Users\Sam\miniconda3\lib\site-packages\torch\include\pybind11\cast.h(1429): error: too few arguments for template template parameter "Tuple"
          detected during instantiation of class "pybind11::detail::tuple_caster<Tuple, Ts...> [with Tuple=std::pair, Ts=<T1, T2>]"
(1507): here

C:\Users\Sam\miniconda3\lib\site-packages\torch\include\pybind11\cast.h(1503): error: too few arguments for template template parameter "Tuple"
          detected during instantiation of class "pybind11::detail::tuple_caster<Tuple, Ts...> [with Tuple=std::pair, Ts=<T1, T2>]"
(1507): here

C:\Users\Sam\miniconda3\envs\wisp\kaolin-wisp\wisp\csrc\external\mesh2sdf_kernel.cu(592): warning #177-D: variable "split_factor" was declared but never referenced

2 errors detected in the compilation of "wisp/csrc/external/mesh2sdf_kernel.cu".
error: command 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.6\\bin\\nvcc.exe' failed with exit code 1

I was wondering if anybody managed to make the library work properly with the GUI on Windows 10, and if so if they could provide the versions of the libraries installed and the commit from which to pull the code.
Thanks in advance for any help.

other system informations:
OS Name Microsoft Windows 10 Home
Version 10.0.19045 Build 19045
gpu 3050Ti (Laptop)
NVIDIA-SMI 511.23 Driver Version: 511.23 CUDA Version: 11.6
nvcc Build cuda_11.6.r11.6/compiler.30794723_0

App default channels should be initialized from scene graph

The app's default channels to visualize should be auto-loaded from the channels supported by objects in the scene graph

Issues on Kaolin & Cython version

The kaolin-wisp library is very exciting!

However, during installation I encountered several problems.

First, it seems kaolin-wisp requires kaolin 0.12.0, since if it is version 0.11.0 an error about kaolin camera class will occur.

Second, when installing kaolin it constantly interrupt the installation since my Cython version doesn't match the specified version exactly.
However, my Cython version is "0.29.29" and the specified version is "0.29.20", which I think there are no major differences and should not be a blocker.
My solution is to comment out the version checker in the setup.py

ImportError: cannot import name 'parse_options' from 'wisp.config_parser'

Obtaining this error while trying to test NGLOD:

(wisp) PS E:\Projects\kaolin-wisp> python app\main.py --config configs\nglod_nerf.yaml --dataset-path .\data\V8 --dataset-num-workers 4
Traceback (most recent call last):
  File "app\main.py", line 14, in <module>
    from wisp.config_parser import parse_options, argparse_to_str, get_modules_from_config, \
ImportError: cannot import name 'parse_options' from 'wisp.config_parser' (e:\projects\kaolin-wisp\wisp\config_parser.py)

Devcontainer

It's very useful to have the docker images, especially for development. It's great to quickly jump in the container and develop. However, since wisp comes bundled, it's hard to develop from within the container.

Is there an idiomatic way of incorporating dynamic changes to the original wisp python code while in the container, in a persistent fashion that does not involve rebuilding the container? E.g., have the python part of wisp being mounted as a volume.

Error Training with Hashtable

Hi thank you for the awsome work!

Using Hashgrid for the fox scene from [Instant-NGP] repository with the new version of the app scripts returns me this error

Seem a problem with the conversion from half to float.

Thank you again

Mesh-to-octree conversion consumes a lot of memory

The mesh_to_octree() function in ops/spc/conversions.py has a number of surface samples hardcoded to 100M for voxel initialization. This consumes a large amount of memory, so much that even on a half-decent GPU like a Titan V (12GB), the vanilla NGLOD-SDF config will fail on a simple Armadillo.

One way to address this would be to expose this number directly in a config file (at the risk of possibly undersampling the shape). Another option would be to catch the runtime error to reduce the load to something more reasonable (e.g., 50M), which could be handled by a simple split-concat:

try:
   samples = mesh_ops.sample_surface(vertices.cuda(), faces.cuda(), 100000000)[0]
except RuntimeError as e:
   samples = [mesh_ops.sample_surface(vertices.cuda(), faces.cuda(), 10000000)[0] for _ in range(5)]
   samples = torch.cat(samples, dim=0)
   # Mention something about reducing number of samples to fit memory

Anyway, just a tip to make this application a bit more accessible to newcomers :)

octree creation consumes too much memory

Thanks again for the amazing library!

I've been using it with other students for a project for my master degree. The problem is that as it's usual for students, we don't have super powerfull machines, and we had to use laptops with 2 or 4GB of VRAM.

I know these kinf of machines don't offer the best experience possible and even simple runs will take a lot of times but they should at least work, at least lowering a lot the quality.

The problem is that most of the grids in the library require octrees, and the kaolin implementation for creating the octrees is highly inefficient in terms of space.

Digging a bit deeper I think I think the cause is in ops/conversions/mesh_to_spc/mesh_to_spc.cpp from kaolin

at::Tensor points_to_octree(
    at::Tensor points,
    uint32_t level) {
#ifdef WITH_CUDA
    uint32_t psize = points.size(0);
    at::Tensor morton = at::zeros({KAOLIN_SPC_MAX_POINTS}, points.options().dtype(at::kLong));
    at::Tensor info = at::zeros({KAOLIN_SPC_MAX_POINTS}, points.options().dtype(at::kInt));
    at::Tensor psum = at::zeros({KAOLIN_SPC_MAX_POINTS}, points.options().dtype(at::kInt));
    at::Tensor octree = at::zeros({KAOLIN_SPC_MAX_OCTREE}, points.options().dtype(at::kByte));
    at::Tensor pyramid = at::zeros({2, level+2}, at::device(at::kCPU).dtype(at::kInt));
  
    point_data* d_points = reinterpret_cast<point_data*>(points.data_ptr<short>());
    morton_code* d_morton = reinterpret_cast<morton_code*>(morton.data_ptr<int64_t>());
    uint32_t*  d_info = reinterpret_cast<uint32_t*>(info.data_ptr<int>());
    uint32_t*  d_psum = reinterpret_cast<uint32_t*>(psum.data_ptr<int>());
    uchar* d_octree = octree.data_ptr<uchar>();
    int*  h_pyramid = pyramid.data_ptr<int>();
    void* d_temp_storage = NULL;
    uint64_t temp_storage_bytes = GetStorageBytes(d_temp_storage, d_morton, d_morton, KAOLIN_SPC_MAX_POINTS);
    at::Tensor temp_storage = at::zeros({(int64_t)temp_storage_bytes}, points.options().dtype(at::kByte));
    d_temp_storage = (void*)temp_storage.data_ptr<uchar>();
    
    uint32_t osize = PointToOctree(d_points, d_morton, d_info, d_psum, d_temp_storage, temp_storage_bytes,
            d_octree, h_pyramid, psize, level);

    return octree.index({Slice(KAOLIN_SPC_MAX_OCTREE - osize, None)});
#else
  AT_ERROR("points_to_octree not built with CUDA");
#endif
}

that initializes the buffer tensors at the maximum size regardless of the size of the input and of the available size, making our low memory gpus crash OOM (it triees to initialize a few GB of memory).

In order to make it work on our laptops we made two changes in wisp/accelstructs/octree_as.py, :

modify the make_dense function (in order to be able to create the octree at the beginning), that was calling points_to_octree without any reason since we already know the size of the octree and it's filled with 255.

    @classmethod
    def make_dense(cls, level) -> OctreeAS:
        """ Builds the acceleration structure and initializes full occupancy of all cells.

        Args:
            level (int): The depth of the octree.
        """
        t=0
        for i in range(level):
            t+=(2**i)**3
        octree = torch.ones(t, dtype=torch.uint8, device='cuda')*255
        #octree = wisp_spc_ops.create_dense_octree(level)
        
        return OctreeAS(octree)

modify from_quantized_points (in oder to be able to prune the structure), here we just plainly had to rewrite the function in a way that didn't crash the gpu

import numpy as np
def quantized_to_octree(quantized_points, level):

    if quantized_points.device!='cpu':
        quantized_points=quantized_points.detach().cpu()

    pts=np.array(torch.unique(quantized_points,dim=0), dtype=np.uint8)[:,None]
    bits = np.unpackbits(pts, 1)[...,[0,1,2]]
    bits = bits.reshape(len(bits),-1)

    [email protected](2,np.arange(24)[ : :-1])

    oct=np.zeros((2**level)**3, dtype=bool)
    oct[m_idx]=1
    octree=[]

    for _ in range(level):
        octL = np.packbits(oct.reshape(-1,8),-1,bitorder='little')
        oct = octL>0
        octree.insert(0,octL[oct])

    return torch.tensor(np.concatenate(octree), dtype=torch.uint8, device='cuda')

...

   def from_quantized_points(cls, quantized_points, level) -> OctreeAS:
        """ Builds the acceleration structure from quantized (integer) point coordinates.

        Args:
            quantized_points (torch.LongTensor): 3D coordinates of shape [num_coords, 3] in
                                                 integer coordinate space [0, 2**level]
            level (int): The depth of the octree.
        """
        #octree = spc_ops.unbatched_points_to_octree(quantized_points, level, sorted=False)
        octree = quantized_to_octree(quantized_points, level)
        return OctreeAS(octree

we had to use numpy because torch doesn't have an equivalent for packbits and unpackbits.

The implementation of make_dense should be pretty close to optimal (unless there is a closed form formula to compute the number of elementr of the octree, but the improvement is marginal).
The implementation of from_quantized_points is definitely not optimal, a custom cuda kernel would do the job way better, but I don't have the time at the moment to write one, and I probably don't have enough experience in cuda to optimize it properly.

I'm not sure if the implementation done in that way was done to achive maximum speed at the expense of memory, but there should be an option to use a low memory
algorithm when there is not enough VRAM.
I don't know if you are also in touch with the kaolin team, and also to them the problem, but would be a nice to have if there was a flag to turn on low memory octree computation.

Hope I helped sombody

Also, I'm not really used to github yet, while we were working of the project we realized there were several small things we would have changed in order to make the library more flexible or more efficient. Should I make a different post for each one of them or can I just dump them in a single issue? or should I use pull requests?

Thanks again for your time

How to set the config file to use the SPCField for training ?

How to find the VQAD model size correctly

Thanks for this great work!
I am trying to reproduce the VQAD paper results.
I train the VQAD model using the RTMV datasets with default config.
But the model size is over 20MB, not small as the paper.
Could you please show me how to calculate the correct model size?

training config

global:
  exp_name: "test-vqad-nerf"

optimizer:
  optimizer_type: "rmsprop"
  lr: 0.001

dataset:
  dataset_type: "multiview"
  num_rays_sampled_per_img: 4096
  multiview_dataset_format: "rtmv"
  mip: 2
  bg_color: "white"

renderer:
  tracer_type: "PackedRFTracer"
  num_steps: 16
  render_batch: 4000
  camera_origin:
    - -3.0
    - 0.65
    - -3.0
  shading_mode: "rb"
  render_res:
    - 1024
    - 1024

trainer:
  trainer_type: "MultiviewTrainer"
  epochs: 50
  batch_size: 1
  model_format: "full"
  valid_every: 50
  save_every: 50
  render_every: 50

grid:
  grid_type: "CodebookOctreeGrid"
  interpolation_type: "linear"
  multiscale_type: "sum"
  feature_dim: 5
  feature_std: 0.01
  base_lod: 5
  num_lods: 4
  codebook_bitwidth: 4

net:
  nef_type: "NeuralRadianceField"
  hidden_dim: 128
  num_layers: 1
  out_dim: 4

embedder:
  embedder_type: "positional"

the script I used to calculate the model size

import torch

model = torch.load("path/to/model")
param_size = 0

for param in model.parameters():
    param_size += param.nelement() * param.element_size()
buffer_size = 0
for buffer in model.buffers():
    buffer_size += buffer.nelement() * buffer.element_size()

size_all_mb = (param_size + buffer_size) / 1024**2
print('model size: {:.3f}MB'.format(size_all_mb))

Thanks in advance!

Can kaolin-wisp run on Windows?

As the title. Thank you very much!

Does wisp._C.ops.hashgrid_interpolate_cuda provide gradients w.r.t. coordinates?

I am trying to enhance NeuS with a hash grid encoding. NeuS utilizes the normal of the SDF as input to its color field, and this normal is computed as the gradient of the SDF w.r.t. the coordinates. That means that if I implement the SDF field with a hash grid, I will need to be able to differentiate the hash grid not only w.r.t. the features, but also the coordinates. It looks like this is not currently supported in kaolin-wisp nor in kaolin, as I am getting None when I try to obtain the gradient w.r.t. the input coordinates.

Is it planned to add this capability to kaolin-wisp?

IndexError: tensors used as indices must be long, byte or bool tensors

Hi,

I have been trying to run kaolin wisp on the lego synthetic data but I have this bug.


python app/nerf/main_nerf.py --dataset-path /media/phong/NEW_DATA/data/nerf_synthetic/hotdog/ --config app/nerf/configs/nerf_triplanar.yaml 
loading data: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:01<00:00, 78.85it/s]
/home/phong/miniconda3/envs/wisp/lib/python3.9/site-packages/torch/functional.py:478: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2894.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
2023-02-06 12:58:56,427|    INFO| Active LODs: [5, 6, 7, 8]
2023-02-06 12:58:56,434|    INFO| # Feature Vectors: 3168144
2023-02-06 12:58:56,459|    INFO| Info: 
```{ 'dataset': { 'bg_color': 'black',
               'dataloader_num_workers': 0,
               'dataset_num_workers': -1,
               'dataset_path': '/media/phong/NEW_DATA/data/nerf_synthetic/hotdog/',
               'mip': 2,
               'multiview_dataset_format': 'standard',
               'num_rays_sampled_per_img': 4096},
  'grid': { 'base_lod': 5,
            'blas_level': 7,
            'blas_type': 'octree',
            'codebook_bitwidth': 8,
            'feature_bias': 0.0,
            'feature_dim': 4,
            'feature_std': 0.01,
            'grid_type': 'TriplanarGrid',
            'interpolation_type': 'linear',
            'max_grid_res': 2048,
            'min_grid_res': 16,
            'multiscale_type': 'sum',
            'num_lods': 4,
            'prune_density_decay': 0.6,
            'prune_min_density': 2.956033378250884,
            'tree_type': 'geometric'},
  'logging': {'exp_name': 'test-nerf', 'log_level': 20, 'perf': False},
  'nef': { 'activation_type': 'relu',
           'hidden_dim': 64,
           'layer_type': 'none',
           'num_layers': 1,
           'pos_embedder': 'none',
           'pos_multires': 10,
           'position_input': False,
           'view_embedder': 'positional',
           'view_multires': 4},
  'optimizer': { 'eps': 1e-08,
                 'grid_lr_weight': 100.0,
                 'lr': 0.001,
                 'optimizer_type': 'rmsprop',
                 'rgb_loss': 1.0,
                 'weight_decay': 0},
  'optional arguments': { 'config': 'app/nerf/configs/nerf_triplanar.yaml',
                          'help': None,
                          'profile': False},
  'positional arguments': {},
  'renderer': { 'camera_clamp': [0, 10],
                'camera_fov': 30,
                'camera_lookat': [0, 0, 0],
                'camera_origin': [-3.0, 0.65, -3.0],
                'camera_proj': 'persp',
                'render_batch': 4000,
                'render_res': [1024, 1024]},
  'tracer': {'num_steps': 512, 'raymarch_type': 'voxel'},
  'trainer': { 'batch_size': 1,
               'epochs': 50,
               'grow_every': -1,
               'growth_strategy': 'increase',
               'log_dir': '_results/logs/runs/',
               'log_tb_every': 5,
               'model_format': 'full',
               'only_last': False,
               'pretrained': None,
               'prune_every': -1,
               'random_lod': False,
               'render_tb_every': -1,
               'resample': False,
               'resample_every': 1,
               'save_as_new': False,
               'save_every': -1,
               'valid_every': -1,
               'valid_only': False,
               'wandb_entity': None,
               'wandb_project': None,
               'wandb_run_name': None,
               'wandb_viz_nerf_angles': 20,
               'wandb_viz_nerf_distance': 3}}```
2023-02-06 12:58:56,460|    INFO| Training on /media/phong/NEW_DATA/data/nerf_synthetic/hotdog/
2023-02-06 12:58:56,460|    INFO| Total number of parameters: 1065095
2023-02-06 12:58:56,460|    INFO| Using NVIDIA GeForce RTX 2080 Ti with CUDA v11.3
2023-02-06 12:58:56,464|    INFO| No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
[i] Using PYGLFW_IMGUI (GL 2.1)
2023-02-06 12:58:57,224|    INFO| [i] Using PYGLFW_IMGUI (GL 2.1)
[i] Running at 60 frames/second
2023-02-06 12:58:57,236|    INFO| [i] Running at 60 frames/second
/home/phong/miniconda3/envs/wisp/lib/python3.9/site-packages/kaolin/render/spc/raytrace.py:121: UserWarning: mark_first_hit has been deprecated, please use mark_pack_boundaries instead
  warnings.warn("mark_first_hit has been deprecated, please use mark_pack_boundaries instead")
Traceback (most recent call last):
  File "/home/phong/data/Work/Code/kaolin-wisp/app/nerf/main_nerf.py", line 494, in <module>
    app.run()  # Run in interactive mode
  File "/home/phong/data/Work/Code/kaolin-wisp/wisp/renderer/app/wisp_app.py", line 243, in run
    app.run()   # App clock should always run as frequently as possible (background tasks should not be limited)
  File "/home/phong/miniconda3/envs/wisp/lib/python3.9/site-packages/glumpy/app/__init__.py", line 362, in run
    run(duration, framecount)
  File "/home/phong/miniconda3/envs/wisp/lib/python3.9/site-packages/glumpy/app/__init__.py", line 344, in run
    count = __backend__.process(dt)
  File "/home/phong/miniconda3/envs/wisp/lib/python3.9/site-packages/glumpy/app/window/backends/backend_glfw_imgui.py", line 448, in process
    window.dispatch_event('on_draw', dt)
  File "/home/phong/miniconda3/envs/wisp/lib/python3.9/site-packages/glumpy/app/window/event.py", line 396, in dispatch_event
    if getattr(self, event_type)(*args):
  File "/home/phong/data/Work/Code/kaolin-wisp/wisp/renderer/app/wisp_app.py", line 519, in on_draw
    self.render()     # Render objects uploaded to GPU
  File "/home/phong/miniconda3/envs/wisp/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/phong/data/Work/Code/kaolin-wisp/wisp/renderer/app/wisp_app.py", line 487, in render
    img, depth_img = self.render_canvas(self.render_core, dt, self.canvas_dirty)
  File "/home/phong/data/Work/Code/kaolin-wisp/wisp/renderer/app/wisp_app.py", line 378, in render_canvas
    renderbuffer = render_core.render(time_delta, force_render)
  File "/home/phong/miniconda3/envs/wisp/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/phong/data/Work/Code/kaolin-wisp/wisp/renderer/core/render_core.py", line 210, in render
    rb = self._render_payload(payload, force_render)
  File "/home/phong/data/Work/Code/kaolin-wisp/wisp/renderer/core/render_core.py", line 309, in _render_payload
    rb = renderer.render(in_rays)
  File "/home/phong/data/Work/Code/kaolin-wisp/wisp/renderer/core/renderers/radiance_pipeline_renderer.py", line 55, in render
    rb += self.tracer(self.nef,
  File "/home/phong/miniconda3/envs/wisp/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/phong/data/Work/Code/kaolin-wisp/wisp/tracers/base_tracer.py", line 154, in forward
    rb = self.trace(nef, rays, requested_channels, requested_extra_channels, **input_args)
  File "/home/phong/data/Work/Code/kaolin-wisp/wisp/tracers/packed_rf_tracer.py", line 125, in trace
    ridx_hit = ridx[boundary]
IndexError: tensors used as indices must be long, byte or bool tensors

Whats the difference between cameras and image planes gizmos in gui?

Whats the difference between cameras and image planes in gui?
I would love to see image planes but they look pretty much the same to me now?
Just checking i dont miss anything before making myself some

How to run it on WSL?

I prepared and compiled everything but it triggers errors when trying to load it under wsl.
It starts loading the dataset (i'm trying with the default fox) but then it stops

Install procedure directions

In the most recent commit 351a04a the install instructions suggested:
pip install kaolin==0.12.0 -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-${TORCH_VER}_cu${CUDA_VER}.html

When trying this, it seems like kaolin released 0.11.0 but not 0.12.0 yet? So I assume we need to install manually from source?
https://pypi.org/project/kaolin/#history

pycuda make default context error

I am trying to install the library. I have the following configurations:
Ubuntu 22.04.1 LTS
Nvidia 3090

When I run the script for interactive rendering, I get the following error:

        [i] Using PYGLFW_IMGUI (GL 2.1)
        Traceback (most recent call last):
          File "app/main_interactive.py", line 12, in <module>
            setup_cuda_context()     # Must be called before any torch operations take place
          File "/kaolin-wisp/app/cuda_guard.py", line 34, in setup_cuda_context
            import pycuda.gl.autoinit
          File "kaolin-wisp/pycuda/pycuda/gl/autoinit.py", line 10, in <module>
            context = make_default_context(lambda dev: cudagl.make_context(dev))
          File "kaolin-wisp/pycuda/pycuda/tools.py", line 226, in make_default_context
            raise RuntimeError(
        RuntimeError: make_default_context() wasn't able to create a context on any of the 1 detected devices

typo

kaolin-wisp/wisp/models/grids/triplanar_grid.py

Line 76 in 4d93ade

align_corners=True, padding_modes=self.padding_mode)[0,:,:,0].transpose(0,1)

should be mode not modes

Wisp supports joint rendering of various pipelines (NeRF, SDFs, meshes)

Very curious if there are any examples of combining pipelines? Just add them to multiple scene_graph.neural_pipelines

Installation without libopenexr-dev

Is it possible to install Kaolin and Kaolin Wisp on Ubuntu without OpenEXR? I'm unable to install libopenexr-dev on a server without sudo access. Thanks!

merge 2 octrees?

I have a case where i want to merge 2 separate octrees, just wondering if this already possible?

Unknown backend (glfw_imgui)

I was wondering where to find glfw_imgui backend?

[x] Unknown backend (glfw_imgui)
2022-08-10 08:23:28,895|CRITICAL| [x] Unknown backend (glfw_imgui)
p: init.py
[x] Available backends are: ('glfw', 'pyglet', 'sdl', 'sdl2', 'osxglut', 'freeglut', 'qt5', 'pyside', 'pyimgui')

Clarify licensing please

hi,
After hearing from other people that they havent heard back from their licensing request either i am concerned.
Could you please clarify the licensing conditions for this great library? Hate to be that person but it matters...

Interactive training - OpenGL issue

Hello,

Thanks for the great work!
I ran into an issue that is actually very similar to #13. But this issue was closed, and it seems that some other people still have the same problem. Because I'm running on WSL2 (see below), and the previous one was closed, I opened a second issue.

Setup

I'm using WSL2 with Ubuntu 20.04 on my machine, with an RTX 3080 (Driver version 511.65, CUDA version 11.6).

Method

I used your Dockerfile to build the Wisp Docker image.
I'm running the container by forwarding the DISPLAY environment variable and using VcxSrv as my X server.
I confirmed that interactive windows could be rendered, using the glxgears demo from mesa-utils.

Issue

Running the interactive training leads to this issue:

Traceback (most recent call last):
  File "/wisp/app/main_interactive.py", line 12, in <module>
    setup_cuda_context()     # Must be called before any torch operations take place
  File "/wisp/app/cuda_guard.py", line 29, in setup_cuda_context
    window = app.Window(width=10, height=10, title='dummy', visible=False)
  File "/opt/conda/lib/python3.9/site-packages/glumpy/app/__init__.py", line 188, in __new__
    window = __backend__.Window(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/glumpy/app/window/backends/backend_glfw_imgui.py", line 248, in __init__
    self.imguiRenderer = GlfwRenderer(self._native_window, attach_callbacks=False)
  File "/opt/conda/lib/python3.9/site-packages/imgui/integrations/glfw.py", line 13, in __init__
    super(GlfwRenderer, self).__init__()
  File "/opt/conda/lib/python3.9/site-packages/imgui/integrations/opengl.py", line 60, in __init__
    super(ProgrammablePipelineRenderer, self).__init__()
  File "/opt/conda/lib/python3.9/site-packages/imgui/integrations/base.py", line 17, in __init__
    self._create_device_objects()
  File "/opt/conda/lib/python3.9/site-packages/imgui/integrations/opengl.py", line 108, in _create_device_objects
    self._attrib_location_tex = gl.glGetUniformLocation(self._shader_handle, "Texture")
  File "src/latebind.pyx", line 51, in OpenGL_accelerate.latebind.Curry.__call__
  File "/opt/conda/lib/python3.9/site-packages/OpenGL/GL/VERSION/GL_2_0.py", line 430, in glGetUniformLocation
    return baseOperation( program, name )
  File "/opt/conda/lib/python3.9/site-packages/OpenGL/platform/baseplatform.py", line 415, in __call__
    return self( *args, **named )
  File "src/errorchecker.pyx", line 58, in OpenGL_accelerate.errorchecker._ErrorChecker.glCheckError
OpenGL.error.GLError: GLError(
        err = 1282,
        baseOperation = glGetUniformLocation,
        cArguments = (1, b'Texture\x00'),
        result = -1
)

Even simpler, running app-simple.py from the glumpy example, with the additional line app.use("glfw_imgui") reaches the same OpenGL error. Maybe I should directly open an issue on the glumpy repository, but other kaolin-wisp users might face the same issue.
If you have time to provide some help on this, it would be awesome!

Training time and performance

Thanks for opening source such a huge & great project!
I would like to compare to other implementations, do you have any benchmarks on public datasets (e.g. synthetic-nerf)?
Btw, this is what I get from 5 minutes of training on fox using this command:

python3 app/main_interactive.py --config configs/ngp_nerf_interactive.yaml --multiview-dataset-format standard --mip 0 --dataset-path ../instant-ngp/data/nerf/fox/

I don't see any PSNR on the graph, but judging from the visual quality and the loss, I reckon it is far less good than instant-ngp of the same training time. Is it to be expected?

Just saw your teaser image, it took 26 minutes to get to visually good quality as shown?

Request for joining origanization

I'm open source advocate and love to do open source contributions

Interactive Training Crashes Immediately

This might be more of an OpenGL setup issue, but it's only occuring for the interactive rendering. I can use the nerf app fine in headless mode, but when I try to use the GUI I get the following error.

X Error of failed request:  BadWindow (invalid Window parameter)
  Major opcode of failed request:  150 (GLX)
  Minor opcode of failed request:  16 (X_GLXVendorPrivate)
  Resource id in failed request:  0x2c00009
  Serial number of failed request:  0
  Current serial number in output stream:  152

I followed the solution in #66 for modifying the window config for the correct openGL version and that got me a step further. I added

config = app.configuration.Configuration()
config.major_version=3
config.minor_version=2
config.profile='core'
window = app.Window(..., config=config)

to wisp/cuda_guard.py and then I encountered another issue where make_default_context() wasn't able to create a context on any of the 1 detected devices and solved that by including __NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia before running the python script.

Now when I run the script, a transparent window pops up, the data is loaded, and training starts but immediately crashes with the following error:

[i] Using PYGLFW_IMGUI (GL 2.1)
2023-01-11 19:37:33,079|    INFO| [i] Using PYGLFW_IMGUI (GL 2.1)
[i] Running at 60 frames/second
2023-01-11 19:37:33,111|    INFO| [i] Running at 60 frames/second
Traceback (most recent call last):
  File "app/nerf/main_nerf.py", line 490, in <module>
    app.run()  # Run in interactive mode
  File "/home/ubuntu/nr/kaolin-wisp/wisp/renderer/app/wisp_app.py", line 248, in run
    app.run()   # App clock should always run as frequently as possible (background tasks should not be limited)
  File "/opt/conda/envs/wisp/lib/python3.8/site-packages/glumpy/app/__init__.py", line 362, in run
    run(duration, framecount)
  File "/opt/conda/envs/wisp/lib/python3.8/site-packages/glumpy/app/__init__.py", line 344, in run
    count = __backend__.process(dt)
  File "/opt/conda/envs/wisp/lib/python3.8/site-packages/glumpy/app/window/backends/backend_glfw_imgui.py", line 448, in process
    window.dispatch_event('on_draw', dt)
  File "/opt/conda/envs/wisp/lib/python3.8/site-packages/glumpy/app/window/event.py", line 396, in dispatch_event
    if getattr(self, event_type)(*args):
  File "/home/ubuntu/nr/kaolin-wisp/wisp/renderer/app/wisp_app.py", line 527, in on_draw
    self.render()     # Render objects uploaded to GPU
  File "/opt/conda/envs/wisp/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/ubuntu/nr/kaolin-wisp/wisp/renderer/app/wisp_app.py", line 499, in render
    self._blit_to_gl_renderbuffer(img, depth_img, self.canvas_program, self.cuda_buffer,
  File "/home/ubuntu/nr/kaolin-wisp/wisp/renderer/app/wisp_app.py", line 414, in _blit_to_gl_renderbuffer
    canvas_program.draw(gl.GL_TRIANGLE_STRIP)
  File "/opt/conda/envs/wisp/lib/python3.8/site-packages/glumpy/gloo/program.py", line 603, in draw
    self.activate()
  File "/opt/conda/envs/wisp/lib/python3.8/site-packages/glumpy/gloo/globject.py", line 95, in activate
    self._activate()
  File "/opt/conda/envs/wisp/lib/python3.8/site-packages/glumpy/gloo/program.py", line 393, in _activate
    attribute.activate()
  File "/opt/conda/envs/wisp/lib/python3.8/site-packages/glumpy/gloo/globject.py", line 95, in activate
    self._activate()
  File "/opt/conda/envs/wisp/lib/python3.8/site-packages/glumpy/gloo/variable.py", line 383, in _activate
    gl.glVertexAttribPointer(self.handle, size, gtype, gl.GL_FALSE, stride, offset)
  File "/opt/conda/envs/wisp/lib/python3.8/site-packages/OpenGL/latebind.py", line 63, in __call__
    return self.wrapperFunction( self.baseFunction, *args, **named )
  File "/opt/conda/envs/wisp/lib/python3.8/site-packages/OpenGL/GL/VERSION/GL_2_0.py", line 470, in glVertexAttribPointer
    return baseOperation(
  File "/opt/conda/envs/wisp/lib/python3.8/site-packages/OpenGL/latebind.py", line 43, in __call__
    return self._finalCall( *args, **named )
  File "/opt/conda/envs/wisp/lib/python3.8/site-packages/OpenGL/wrapper.py", line 1392, in wrapperCall
    raise err
  File "/opt/conda/envs/wisp/lib/python3.8/site-packages/OpenGL/wrapper.py", line 1385, in wrapperCall
    result = wrappedOperation( *cArguments )
  File "/opt/conda/envs/wisp/lib/python3.8/site-packages/OpenGL/error.py", line 230, in glCheckError
    raise self._errorClass(
OpenGL.error.GLError: GLError(
	err = 1282,
	description = b'invalid operation',
	baseOperation = glVertexAttribPointer,
	pyArgs = (
		0,
		2,
		GL_FLOAT,
		GL_FALSE,
		16,
		c_void_p(None),
	),
	cArgs = (
		0,
		2,
		GL_FLOAT,
		GL_FALSE,
		16,
		c_void_p(None),
	),
	cArguments = (
		0,
		2,
		GL_FLOAT,
		GL_FALSE,
		16,
		c_void_p(None),
	)
)

Has anyone else encountered this?

Setup:

Ubuntu 20.04.5 running remotely and connected over RDP
CUDA 11.7 / Driver 515.65.01
glxinfo -B produces

name of display: :10.0
display: :10  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
   Vendor: Mesa/X.org (0xffffffff)
   Device: llvmpipe (LLVM 12.0.0, 256 bits) (0xffffffff)
   Version: 21.2.6
   Accelerated: no
   Video memory: 61287MB
   Unified memory: no
   Preferred profile: core (0x1)
   Max core profile version: 4.5
   Max compat profile version: 3.1
   Max GLES1 profile version: 1.1
   Max GLES[23] profile version: 3.2
OpenGL vendor string: Mesa/X.org
OpenGL renderer string: llvmpipe (LLVM 12.0.0, 256 bits)
OpenGL core profile version string: 4.5 (Core Profile) Mesa 21.2.6
OpenGL core profile shading language version string: 4.50
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile

OpenGL version string: 3.1 Mesa 21.2.6
OpenGL shading language version string: 1.40
OpenGL context flags: (none)

OpenGL ES profile version string: OpenGL ES 3.2 Mesa 21.2.6
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20

glxgears works completely fine. I can also get the instant-ngp GUI up, but I'm unable to interactively train a model there for other reasons.

model.pth How to load ?

More of a question than an issue.
I see that the app saves "model.pth" which by the looks of it contains the pipeline definition and the trained weights, etc...

Is there an example on how to reload that ? (for instance start the interactive app with it ?)
Thanks

Julien

How to convert nerf model to mesh

First, thank you so much for this library! I am wondering is there yet a function that can convert a trained nerf model to the mesh (like the one in instant-ngp)? I look into the code and could not find it.

How to reproduce the results in the paper (instant-ngp)?

I train the ngp-nerf with conmand
python3 app/main.py --config configs/ngp_nerf.yaml --multiview-dataset-format standard --mip 0 --dataset-path data/lego
I only changed epochs to 100
The PSNR of result is only 24.52 in the lego dataset.
How can i reproduce the results in the paper (instant-ngp)

version `GLIBCXX_3.4.30' not found in docker image

Thanks for this amazing library!

Let me ask about running example application on docker image.
I created an docker image following this documentation and tried to run NGLOG-NeRF with:

python3 app/main.py --config configs/nglod_nerf.yaml --dataset-path ./datasets/V8 --dataset-num-workers 4

(I mounted V8 data to /wisp/datasets)

Then I got this error message

Traceback (most recent call last):
  File "/wisp/app/main.py", line 13, in <module>
    from wisp.trainers import *
  File "/wisp/wisp/__init__.py", line 9, in <module>
    from . import ops
  File "/wisp/wisp/ops/__init__.py", line 10, in <module>
    from . import grid
  File "/wisp/wisp/ops/grid.py", line 10, in <module>
    from kaolin import _C
  File "/kaolin/kaolin/__init__.py", line 1, in <module>
    from . import io
  File "/kaolin/kaolin/io/__init__.py", line 5, in <module>
    from . import render
  File "/kaolin/kaolin/io/render.py", line 23, in <module>
    from ..render.camera import generate_perspective_projection
  File "/kaolin/kaolin/render/__init__.py", line 2, in <module>
    from . import mesh
  File "/kaolin/kaolin/render/mesh/__init__.py", line 1, in <module>
    from .utils import *
  File "/kaolin/kaolin/render/mesh/utils.py", line 22, in <module>
    from ... import ops
  File "/kaolin/kaolin/ops/__init__.py", line 8, in <module>
    from . import voxelgrid
  File "/kaolin/kaolin/ops/voxelgrid.py", line 18, in <module>
    from scipy import ndimage
  File "/opt/conda/lib/python3.9/site-packages/scipy/__init__.py", line 155, in <module>
    from . import fft
  File "/opt/conda/lib/python3.9/site-packages/scipy/fft/__init__.py", line 79, in <module>
    from ._helper import next_fast_len
  File "/opt/conda/lib/python3.9/site-packages/scipy/fft/_helper.py", line 3, in <module>
    from ._pocketfft import helper as _helper
  File "/opt/conda/lib/python3.9/site-packages/scipy/fft/_pocketfft/__init__.py", line 3, in <module>
    from .basic import *
  File "/opt/conda/lib/python3.9/site-packages/scipy/fft/_pocketfft/basic.py", line 6, in <module>
    from . import pypocketfft as pfft
ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /opt/conda/lib/python3.9/site-packages/scipy/fft/_pocketfft/pypocketfft.cpython-39-x86_64-linux-gnu.so)

Do you have any ideas about this error?
I'm using ubuntu 22.04 as a host machine

Thanks in advance

	transforms = sorted(glob.glob(os.path.join(root, "*.json")))

	transform_dict = {}

	train_only = False

	if mip is None:
	mip = 0

	if len(transforms) == 1:
	transform_dict['train'] = transforms[0]
	train_only = True
	elif len(transforms) == 3:
	fnames = [os.path.basename(transform) for transform in transforms]

	# Create dictionary of split to file path, probably there is simpler way of doing this
	for _split in ['test', 'train', 'val']:
	for i, fname in enumerate(fnames):
	if _split in fname:
	transform_dict[_split] = transforms[i]
	else:
	assert False and "Unsupported number of splits, there should be ['test', 'train', 'val']"

	if split not in transform_dict:
	assert False and f"Split type ['{split}'] unsupported in the dataset provided"

	def load_nef(args, grid):
	if args.nef_type == 'NeuralRadianceField':
	return NeuralRadianceField(
	grid=grid,
	pos_embedder=args.embedder_type,
	view_embedder=args.embedder_type,
	pos_multires=args.pos_multires,
	position_input=args.position_input,
	view_multires=args.view_multires,
	activation_type=args.activation_type,
	layer_type=args.layer_type,
	hidden_dim=args.hidden_dim,
	num_layers=args.num_layers,
	prune_density_decay=args.prune_density_decay,
	prune_min_density=args.prune_min_density
	)
	elif args.nef_type == 'NeuralSDF':
	return NeuralSDF(
	grid=grid,
	pos_embedder=args.embedder_type,
	pos_multires=args.pos_multires,
	position_input=args.position_input,
	activation_type=args.activation_type,
	layer_type=args.layer_type,
	hidden_dim=args.hidden_dim,
	num_layers=args.num_layers
	)
	else:
	raise ValueError(f'Neural field {args.nef_type} is temporarily unsupported by the config parser.')

nvidiagameworks / kaolin-wisp Goto Github PK

kaolin-wisp's People

Contributors

Stargazers

Watchers

Forkers

kaolin-wisp's Issues

Setup

Method

Issue

Recommend Projects

Recommend Topics

Recommend Org