Hey! Great work. I had a couple questions on the surface reconstruction when the camer

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

Camera poses & coordinate system used about neuralangelo HOT 14 OPEN

nvlabs commented on August 16, 2024

Camera poses & coordinate system used

from neuralangelo.

Comments (14)

mli0603 commented on August 16, 2024 3

Hi @smandava98

If you look at the code below, it reads an image and generates pixel indexes to optimize the geometry. What you would do is read the corresponding masks and remove the unwanted pixels. If you have something working, feel free to submit a PR :)

neuralangelo/projects/neuralangelo/data.py

Lines 49 to 85 in 2213bb3

 def __getitem__(self, idx): 

 """Process raw data and return processed data in a dictionary. 

  Args: 

  idx: The index of the sample of the dataset. 

  Returns: A dictionary containing the data. 

  idx (scalar): The index of the sample of the dataset. 

  image (R tensor): Image idx for per-image embedding. 

  image (Rx3 tensor): Image with pixel values in [0,1] for supervision. 

  intr (3x3 tensor): The camera intrinsics of `image`. 

  pose (3x4 tensor): The camera extrinsics [R,t] of `image`. 

  """ 

 # Keep track of sample index for convenience. 

 sample = dict(idx=idx) 

 # Get the images. 

 image, image_size_raw = self.images[idx] if self.preload else self.get_image(idx) 

 image = self.preprocess_image(image) 

 # Get the cameras (intrinsics and pose). 

 intr, pose = self.cameras[idx] if self.preload else self.get_camera(idx) 

 intr, pose = self.preprocess_camera(intr, pose, image_size_raw) 

 # Pre-sample ray indices. 

 if self.split == "train": 

 ray_idx = torch.randperm(self.H * self.W)[:self.num_rays] # [R] 

 image_sampled = image.flatten(1, 2)[:, ray_idx].t() # [R,3] 

 sample.update( 

 ray_idx=ray_idx, 

 image_sampled=image_sampled, 

 intr=intr, 

 pose=pose, 

 ) 

 else: # keep image during inference 

 sample.update( 

 image=image, 

 intr=intr, 

 pose=pose, 

 ) 

 return sample

from neuralangelo.

mli0603 commented on August 16, 2024 2

Yes, feeding masks is definitely an option. We removed this functionality in the release as we did not feel there will be lots of use cases. Usually people are happy with both lamps and house ;)

I will add this to our todo list

from neuralangelo.

mli0603 commented on August 16, 2024 1

Hi @smandava98

Neuralangelo optimizes the surface in a canonical coordinate system, which is the coordinate where camera/object poses are defined.
Two different video sequences of the same object often do NOT have the same coordinate frame due to how COLMAP initializes the world coordinate. However, it is possible to align the coordinates using a single affine transformation.

from neuralangelo.

smandava98 commented on August 16, 2024

Thanks Max. I have a followup question:

How can I run the neural reconstruction around a particular object in a video? For example, I have a video of a house on a road but a street lamp is blocking parts of the house in some frames which are then more visible in future frames. How can I somehow omit this object in the reconstruction and have missing pieces of that house filled up from later video frames?

from neuralangelo.

mli0603 commented on August 16, 2024

Hi @smandava98

Neuralangelo can handle occlusion and will simply reconstruct both objects (i.e., the lamp and the house). If you don't want the lamp, you have two options:

remove the lamp after extracting the mesh using mesh editing softwares (such as meshlab or blender)
remove the lamp from the bounding regions. You can see step 5 in Data Preprocessing on how to adjust the bounding region

from neuralangelo.

smandava98 commented on August 16, 2024

This still seems to be a manual approach. Is there any way to make it automatic? I have a tons of videos of houses but are being obstructed by lamps and I have a instance segmentation model for the lamps that gives me the masks. Is there not a way where I can just feed Neuralangelo a mask that it can ignore automatically?

from neuralangelo.

smandava98 commented on August 16, 2024

Thank you! Adding that option will be extremely helpful. My use case is I want to get clear reconstructions of certain objects and lamps and all that are unnecessary elements in that reconstruction I'm trying to create.

Hope this illuminated the use case better.

from neuralangelo.

smandava98 commented on August 16, 2024

Hi @mli0603. Thanks for your detailed and consistent support to the community. It's amazing.

While I wait for this enhancement to be added, can you please point me to which file(s) I might need to modify to have it ignore a mask? Since you mentioned it was included before but it got removed.

from neuralangelo.

smandava98 commented on August 16, 2024

Awesome, thank you. Will definitely put up a PR if I get it working :)

from neuralangelo.

tkl2020 commented on August 16, 2024

I also encountered the same problem, I added Neus's Mask loss and tried white background, but it didn't work. Ask the author if he has any feasible suggestions.

from neuralangelo.

chenhsuanlin commented on August 16, 2024

Please also see #58 for related discussions.

from neuralangelo.

lishaobingdong commented on August 16, 2024

@smandava98 @tkl2020
Hello!
I also encountered this problem. I read the mask images and added mask_loss, but it seemed to have no effect. What files did you change to succeed?
Thank you! Looking forward to replying

from neuralangelo.

lishaobingdong commented on August 16, 2024

@smandava98 @tkl2020 Hello! I also encountered this problem. I read the mask images and added mask_loss, but it seemed to have no effect. What files did you change to succeed? Thank you! Looking forward to replying

I succeeded by setting background enabled to false and using image=image*mask

from neuralangelo.

Davidyao99 commented on August 16, 2024

@lishaobingdong Hi, I am trying to do the same task of training neuralangelo on a scene with masks to remove certain objects. Can you elaborate a little on how you mange to make it work!! Thank you so much ;))

from neuralangelo.

Camera poses & coordinate system used about neuralangelo HOT 14 OPEN

Comments (14)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	def __getitem__(self, idx):
	"""Process raw data and return processed data in a dictionary.

	Args:
	idx: The index of the sample of the dataset.
	Returns: A dictionary containing the data.
	idx (scalar): The index of the sample of the dataset.
	image (R tensor): Image idx for per-image embedding.
	image (Rx3 tensor): Image with pixel values in [0,1] for supervision.
	intr (3x3 tensor): The camera intrinsics of `image`.
	pose (3x4 tensor): The camera extrinsics [R,t] of `image`.
	"""
	# Keep track of sample index for convenience.
	sample = dict(idx=idx)
	# Get the images.
	image, image_size_raw = self.images[idx] if self.preload else self.get_image(idx)
	image = self.preprocess_image(image)
	# Get the cameras (intrinsics and pose).
	intr, pose = self.cameras[idx] if self.preload else self.get_camera(idx)
	intr, pose = self.preprocess_camera(intr, pose, image_size_raw)
	# Pre-sample ray indices.
	if self.split == "train":
	ray_idx = torch.randperm(self.H * self.W)[:self.num_rays] # [R]
	image_sampled = image.flatten(1, 2)[:, ray_idx].t() # [R,3]
	sample.update(
	ray_idx=ray_idx,
	image_sampled=image_sampled,
	intr=intr,
	pose=pose,
	)
	else: # keep image during inference
	sample.update(
	image=image,
	intr=intr,
	pose=pose,
	)
	return sample