Hi,

Thank you for your excellent work, but I have a question. How should I handle my data so that it can be accepted by this framework? I only have meshes. I do not have the .blender file and the .xml file. And I have texture files of different formart.

Thank you.

GPU memory is not properly freed when switching to other meshes, eventually leading to CUSPARSE_STATUS_ALLOC_FAILED:

Traceback (most recent call last):
  File "scripts/show_largesteps_memory_leak.py", line 16, in <module>
    v = from_differential(M, u, 'Cholesky')
  File "/home/xuzhen/miniconda3/envs/flame/lib/python3.8/site-packages/largesteps/parameterize.py", line 51, in from_differential
    solver = CholeskySolver(L)
  File "/home/xuzhen/miniconda3/envs/flame/lib/python3.8/site-packages/largesteps/solvers.py", line 130, in __init__
    self.solver_1 = prepare(self.L, False, False, True)
  File "/home/xuzhen/miniconda3/envs/flame/lib/python3.8/site-packages/largesteps/solvers.py", line 68, in prepare
    _cusparse.scsrsm2_analysis(
  File "cupy_backends/cuda/libs/cusparse.pyx", line 2103, in cupy_backends.cuda.libs.cusparse.scsrsm2_analysis
  File "cupy_backends/cuda/libs/cusparse.pyx", line 2115, in cupy_backends.cuda.libs.cusparse.scsrsm2_analysis
  File "cupy_backends/cuda/libs/cusparse.pyx", line 1511, in cupy_backends.cuda.libs.cusparse.check_status
cupy_backends.cuda.libs.cusparse.CuSparseError: CUSPARSE_STATUS_ALLOC_FAILED

To reproduce, run this code example with this example mesh (extract armadillo.npz and place it where you run the code below):

import torch
import numpy as np
from tqdm import tqdm

from largesteps.parameterize import from_differential, to_differential
from largesteps.geometry import compute_matrix
from largesteps.optimize import AdamUniform

armadillo = np.load('armadillo.npz')
verts = torch.tensor(armadillo['v'], device='cuda')
faces = torch.tensor(armadillo['f'], device='cuda')

for i in tqdm(range(3000)):
    # assume there's different meshes w/ different topology
    M = compute_matrix(verts, faces, 10)
    u = to_differential(M, verts)
    u.requires_grad_()
    optim = AdamUniform([u], 3e-2)
    for j in range(5):
        v: torch.Tensor = from_differential(M, u, 'Cholesky')
        loss: torch.Tensor = (v.norm(dim=-1) - 1).mean()
        optim.zero_grad()
        loss.backward()
        optim.step()

While running the code above, you should see the GPU memory continuously increase but the expected behavior is that it stays constant.

For example, the result of nvidia-smi dmon -s m while running the code should be something like:

I tried to replace NVDRenderer with mistuba3.0, when I used CholeskySolver, an error popped as:

Exception has occurred: TypeError
solve(): incompatible function arguments. The following argument types are supported:
    1. solve(self, b: tensor[dtype=float32, order='C'], x: tensor[dtype=float32, order='C']) -> None

if i changed the solver to 'CG', the error will be gone, but after several loops, another error will turn up:
Critical Dr.Jit compiler failure: jit_var(r153196083): unknown variable!'
Could anyone help to figure out? thanks.

Hi Baptiste, thanks for publishing the code:-) I found two requirements missing in the installation instructions:

• apt install libeigen3-dev (required for building botsch-kobbelt)
• conda install cudatoolkit-dev -c conda-forge (required by nvdiffrast)

Hi, Thanks for sharing your work!
I tried to use the Tutorial notebook on the Dragon mesh but get really poor results.
Can you share the parameters you used to make the model converge?
thanks a lot

I try to build this project on my windows for a week, and unfortunately failed, can you give me the specific process of build the project on windows? :)
The failure I met is related to libs in the ext/ (mainly numpyeigen).

Not sure if this should be solved here, in cholespy, or nanobind. The from_differential function throws an error if the second argument is a torch.nn.Parameter rather than a tensor. Parameter is directly derived from Tensor, so there's no reason the cast should fail.

TypeError: solve(): incompatible function arguments. The following argument types are supported:
    1. solve(self, b: tensor[dtype=float32, order='C'], x: tensor[dtype=float32, order='C']) -> None

Invoked with types: CholeskySolverF, Parameter, Tensor

It's quite hard to workaround this "from the outside". E.g. doing from_differential(M, x.data) doesn't work because the gradient will be written to x.data.grad whereas the optimizer expects x.grad.

First of all I'd like to thank you for making this phenomenal experience and make it available for testing.
Running the nvdiffrast can be really straight forward and easy for rendering but I'm having some understanding problem with rendering the code inside the blender so please bear with me :)

How do I run the rendering inside the blender? Is it by script editor? [I tried it but it gives me errors]
Is it by running it as command ?

I just need to understand the methodology of rendering that in blender since it's a utility and not included in the Tutorial.

Thank you so much.

I've just cloned the project and I'm trying to run the Tutorial.

The whole directory scenes/suzanne and the suzanne.xml is missing

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-7-8c8db06f0156> in <module>
      1 # Load the scene
      2 filepath = os.path.join(os.getcwd(), "scenes", "suzanne", "suzanne.xml")
----> 3 scene_params = load_scene(filepath)
      4 
      5 # Load reference shape

~/Desktop/large-steps-pytorch/scripts/load_xml.py in load_scene(filepath)
     58     assert ext == ".xml", f"Unexpected file type: '{ext}'"
     59 
---> 60     tree = ET.parse(filepath)
     61     root = tree.getroot()
     62 

~/miniconda3/envs/pytorch3d_06/lib/python3.8/xml/etree/ElementTree.py in parse(source, parser)
   1200     """
   1201     tree = ElementTree()
-> 1202     tree.parse(source, parser)
   1203     return tree
   1204 

~/miniconda3/envs/pytorch3d_06/lib/python3.8/xml/etree/ElementTree.py in parse(self, source, parser)
    582         close_source = False
    583         if not hasattr(source, "read"):
--> 584             source = open(source, "rb")
    585             close_source = True
    586         try:

FileNotFoundError: [Errno 2] No such file or directory: '/home/bobi/Desktop/large-steps-pytorch/scenes/suzanne/suzanne.xml'

fyi you can speed up rendering by ~50%. Replace

vert_light = self.sh.eval(n).contiguous()
light = dr.interpolate(vert_light[None, ...], rast, f)[0]

Hi,

first off, thanks for the code, the package has been working flawlessly for me so far.

I just wanted to report an esoteric issue I have encountered some time ago: in the reproducer below, computing the gradient of the sum over the vertex data produces the following error message

TypeError: solve(): incompatible function arguments. The following argument types are supported:
    1. solve(self, b: tensor[dtype=float32, order='C'], x: tensor[dtype=float32, order='C']) -> None
Invoked with types: cholespy.CholeskySolverF, torch.Tensor, torch.Tensor

whereas computing the mean does not.

In practice, the issue is not super relevant because the vertices are used in downstream computations but you might want to look into that.

from largesteps.parameterize import from_differential, to_differential
from largesteps.geometry import compute_matrix
import torch

device = torch.device('cuda:0')

v = torch.rand((3, 3), dtype=torch.float32, device=device)
f = torch.tensor([[0, 1, 2]], dtype=torch.int32, device=device)

M = compute_matrix(v, f, lambda_=10)

u = to_differential(M, v)
u.requires_grad = True

v = from_differential(M, u, 'Cholesky')

want_to_crash = True

if want_to_crash:
    loss = v.sum()
else:
    loss = v.mean() 

loss.backward() # crashes here if want_to_crash = True

I am on Windows 10, using PyTorch 1.13.1, cholespy 0.1.6, and largesteps 0.2.1.

Hi, I got problems after updating to the latest 0.2.0 version. When my program invoked the from_differential function, it got stuck for a little while, and then exited directly. Nothing (warnings/errors/...) was shown on my prompt, and so I could not figure out what happened. However, the initial 0.1.1 version worked well. Tested on: Windows 10, AMD Ryzen 9 5900HX with Radeon Graphics @ 3.30GHz 16GB, GeForce RTX 3070 Laptop GPU 8GB.

Hi, thanks for the impressive work!

I notice that your recent work on control variates also uses largesteps on 512x256 images. Wondering if it can handle higher resolutions for images and volumes. I tried largesteps on voxel grids up to 64x64x64 but hit a "CHOLMOD error: problem too large" at 128x128x128. The conjugate gradient solver seems to be not as stable as the Cholesky solver.

Is there a more efficient solver for regular lattices (like images and volumes)? Could you share your insights on how to implement such solvers? Your guidance would be greatly appreciated!

Hi，there , I'm here to ask for your help or communicate with you about the issue I have met when I reproduce your fantastic work using our real world data. I fix the light source, but the geometry was a textured mesh with albedo, I want reconstruct the textured geometry from one point of view using your method, Seems like no matter whether I choose the texture to optimize simultaneously or I just only optimize for the mesh vertex position, It's very hard to get the right way for the ideal mesh reconstruction. So would u kindly like to share with me your point about what I have met.

Hi,

Thanks for the great research and sharing the code! I've been using large-steps preconditioning for my project and it's working well. Recently, I worked on improving the performance of my pytorch code and found that changing from pytorch indexing to torch.gather gives me about 30x speed up. I also visualized the computational graph of the part using large-steps preconditioning via torchviz and saw DifferentiableSolveBackward -> IndexBackward0. This indexing seems to come from this repo. I'm pretty satisfied with the speed of the current version but just wondering if it can be further improved by switching to torch.gather.

I compiled and started the project, watches videos and papers but still can't understand the purpose of project. Is it reconstruction from images or this solutions solving one of the problems of area with reconstruction from images?
In sources I can see source and destination model no images. Is this method showing how to get the same model like in target with simple in source but not from images?
Is I understand correctly: The project giving target model -> render it from different positions and using this images for reconstruct the scene back?
If the method using light of areas for reconstruct normals and mesh how far is it from using with photos from real life?

Hi , I am working on baking displacement map via a differentiable learning framework.
And I wonder if the LargeStep method can be used for achieving better results.
However, things come with different since I only want to learn scalar displacement instead of the 3-dim position per vertex.
Generally speaking, is it possible to apply the LargeStep method on more geometry parameters, such as displacement, normal, and/or mix of them?