zju3dv / object_nerf Goto Github PK

Hi, thank you for your great work.
I keep run into segmentation fault when trying to train on Toydesk dataset. It's confusing. Please help.

 python train.py dataset_config=config/toy_desk_2.yml "img_wh=[640,480]" exp_name=my_expr_toydesk_2
  ----------------------------------------
  exp_name: my_expr_toydesk_2
  ckpt_path: null
  dataset_name: toydesk
  img_wh:
  - 640
  - 480
  model:
    use_voxel_embedding: true
    N_freq_xyz: 10
    N_freq_dir: 4
    N_freq_voxel: 6
    D: 8
    W: 256
    skips:
    - 4
    N_scn_voxel_size: 16
    inst_D: 4
    inst_W: 128
    inst_skips:
    - 2
    N_obj_voxel_size: 8
    N_samples: 64
    N_importance: 64
    frustum_bound: -1
    use_disp: false
    perturb: 1
    noise_std: 1
    use_mask: true
    N_vocab: 1000
    N_max_objs: 64
    N_obj_code_length: 64
    N_max_voxels: 800000
  train:
    progressive_train: false
    batch_size: 2048
    chunk: 32768
    num_epochs: 30
    num_gpus: 1
    optimizer: adam
    lr: 0.001
    momentum: 0.9
    weight_decay: 0
    lr_scheduler: poly
    warmup_multiplier: 1
    warmup_epochs: 0
    decay_step:
    - 20
    decay_gamma: 0.1
    poly_exp: 2
    limit_train_batches: 0.05
  prefixes_to_ignore:
  - loss
  loss:
    color_loss_weight: 1.0
    depth_loss_weight: 0.1
    opacity_loss_weight: 10.0
    instance_color_loss_weight: 1.0
    instance_depth_loss_weight: 0.1
  dataset_extra:
    enable_observation_check: false
    max_obs_angle: 40
    max_obs_distance: 3.0
    mask_rebalance_strategy: fg_bg_reweight
    fg_weight: 1.0
    bg_weight: 0.05
    use_bbox: false
    use_bbox_only_for_test: true
    near: 0.8
    far: 24.0
    scale_factor: 16.0
    scene_center:
    - 0.2
    - 1.4
    - 7.1
    train_start_idx: 0
    train_skip_step: 1
    train_max_size: 9999
    validate_idx: 131
    split: datasets/split/our_desk_2_train_0.8
    use_instance_mask: true
    root_dir: data/toydesk/our_desk_2
    bbox_dir: datasets/desk_bbox/desk2/bbox.json
    inst_seg_tag: instance
    val_instance_id: 1
    instance_id:
    - 5
    - 4
    - 2
    - 1
    - 3
    bg_instance_id:
    - 0
    pcd_path: data/toydesk/our_desk_2/pcd_from_mesh.ply
    voxel_size: 0.3
    neighbor_marks: 3
  dataset_config: config/toy_desk_2.yml
  
  ----------------------------------------
  Start with exp_name: 220720_142351_my_expr_toydesk_2.
  Filling the voxel_occupancy...
  Voxel generated: torch.Size([214, 113, 171]) Voxel occupancy ratio: tensor(0.0426, device='cuda:0')
  Voxel used: tensor(176312, device='cuda:0')
  INFO - 2022-07-20 14:23:55,338 - trainer - GPU available: True, used: True
  INFO - 2022-07-20 14:23:55,338 - trainer - TPU available: False, using: 0 TPU cores
  INFO - 2022-07-20 14:23:55,338 - trainer - IPU available: False, using: 0 IPUs
  Training split count 121
  Train idx: 77 -> 115, skip: 1
  Read meta 00119 : 00119 instance 3
  ----------------------------------------
  Valid idx: 131
  INFO - 2022-07-20 14:24:12,671 - gpu - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [2]
  /home/tangxiaotian/miniforge3/envs/object_nerf/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py:623: UserWarning: Checkpoint directory /home/tangxiaotian/object_nerf/logs/220720_142351_my_expr_toydesk_2 exists and is not empty.
    rank_zero_warn(f"Checkpoint directory {dirpath} exists and is not empty.")
  Epoch 0:   0%|                           | 4/904 [00:06<25:35,  1.71s/it, loss=1.04, v_num=0, train/psnr=10.10]段错误 (核心已转储)

assert self.embedding_space_ftr.num_embeddings >= idx_occu.shape[0]
AssertionError

I encounter this error when test on toy_desk dataset. Does anyone know what is causing it?
the error location is in embeding_helper.py line 197.

by the way, how can i change the transformation matrix form slam/opncv format to NDC format, is there any code exist in project that i didnt found.

Hi,

Thanks for the paper and the code contribution. Nice work!

One question: could you please point out the code reference for the "scene-guided occlusion identification (scene guidance + 3D guard mask)" and how they affect the loss backpropagation? Thanks again.

Hi, I am a newbie to NeRF. I would like to know how to handle this dataset?

The description inData Preparation is a bit confusing, do I need to manually convert the dataset from OpenCV format to NDC format? Or is the downloadable dataset already processed?

Also, the download link for scannet_train_detection_data is no longer available, could you please upload it again?

All your help is greatly appreciated.

1. 请问您的tranform_full.json文件是怎么转换得到的，Colmap直接导出的结果的格式似乎不是这样的，。
2. 请问为什么您只对C2w矩阵进行了变换（将YZ轴方向翻转，从OpenCV坐标系变换到opencl坐标系），但是使用PointCloud初始化voxel的时候对PointCloud却没有进行坐标轴的变换呢？(是不是您已经对PointCloud文件进行了这样的处理再保存的？)

I have some error when I tran the model.Should I change the path in the code.
I have the following warnings.
UserWarning: No audio backend is available.
warnings.warn('No audio backend is available.')
UserWarning: torchaudio C++ extension is not available.
warnings.warn('torchaudio C++ extension is not available.')
RPly: Unable to open file
[Open3D WARNING] Read PLY failed: unable to open file: data/toy_desk/our_desk_2/pcd_from_mesh.ply

Hi,
I don't find the edit_type == move.
Can you show me the detailed tutorial for moveing objects?

Hi,

Thanks a lot for releasing this code!

When trying to get it running on my own data, I realized that the data preparation instructions are a bit unclear. If I have a dataset in the LLFF format. Do I need to transform it into the COLMAP format?

Further, is there a way to use the voxel embeddings without knowing the point cloud of the scene a priori?

Thanks!

Hi, Thanks for the code!

I use extract_mesh.py to extract the mesh of the scene branch and I find that the objects also exist in the scene branch. But the Figure 1 in the paper shows that the scene branch only has the background.

So, the scene branch only has "background" or "background + objects"?

Hi,thanks for your great job
but I meet a problem that when i run "python train.py dataset_config=config/scannet_base_0113_multi.yml "img_wh=[640,480]" exp_name=my_expr_scannet_0113" it errors and shows that"FileNotFoundError: [Errno 2] No such file or directory: data/scannet/scans/scene0113_00/scene0113_00.txt'"

in the "scannet_base_0113_multi.yml" i see that " scans_dir: data/scannet/scans".However, i couldn't find a dir named "scans" in the data i download.

Hi, thanks for the nice paper and code!

I have one question about the evaluation setting of the full scene rendering (novel view synthesis). In Table.1 of the paper, quantitative metrics are compared. Are these metrics computed only on the test set, I mean, the frames listed in datasets/split/xxx/test.txt?

Thanks in advance!

Thanks for your great job！
But i have some quesions about the object_nerf/data/toy_desk/bbox/desk1/bbox.json.What's the use of the bbox,and how can i get the bbox.json if i want to use it my own data?

Hi @ybbbbt, I have a simple question: can you please explain the purpose of the parameter "scale_factor" in the ScanNet config files, such as here?

Thank you,
Yash

hi nice work，

Function remove_ Scene_ Object_ By_ IDS seems to be of no use as it did not delete the specified object in the object_nerf]/render_tools
/editable_renderer.py

Hi, I am very interested in this excellent work but I observe that the code is not available now.

May I ask another question: Did you implement your code based on someone else's code? If so, can you tell me where can I get this base code？

Thank you very much, my best wish.

I want to label other scenes, how to get bbox.json?

Hi:
When I train toy_desk datasets, I meet the problem.

 assertError on here
 idx_occu = torch.nonzero(self.voxel_occupancy)
 assert self.embedding_space_ftr.num_embeddings >= idx_occu.shape[0]

 Can you help me solve the problem?

Thank you for sharing the nice work!

Based on the following GitHub issue,
#13
I am trying to train my own model.

However, I noticed that the train split of desk1 data in the Github issue contains image with 95 index,
while the processed images in 'toydesk_data\processed\our_desk_1\full' does not contain "0095.instance.png" and "0095.png".
Can you make a clarification on that?

Thank you.

Hi,

I installed the conda env according to the readme file. But I got this error msg "ModuleNotFoundError: No module named 'torch.ao'" when I try to run training code on toy_desk_2 demo.

Does anyone have the same problem?

Thanks

年都过完了啊哥哥们，速度！

please refresh the link,Thank you!

Hi,
How to determine scale_factor and scene_scater in the config file.
The original value of scale_factor is 5.0, and the value of scene_center is [0.2, 1.4, 7.1].

Hi,

I have successfully completed the training and made a successful demo, your work is awesome. Thank you for your contribution.

However, I noticed in your paper that this algorithm should enable control of individual objects, such as rotating them, adding or removing them from the scene. I would like to experiment with this part, but it is not quite clear which parameters should be modified. Kindly could you please give me some suggestion?

All your help is very much appreciated.

Hello, thanks u for awesome work.

I have some confusion about the Pose matrixs transformation among world, camera, objects. I cannot understand the meaning behind them. I wanna ask if there are some blogs i can refer to ?

Thanks in advance~

Hi~
I wanna ask how do you calculate the values of near, far, scale_factor. I want to mix the your excellent project with another project. But it seems that this two project share different coordinate systems.

Hi,
Thank you for the great job !
I retrain the toydesk model using the toy_desk_?.yml in confg folder. But the test result is much higher than the value in Table 1 ( psnr 25.59 for desk_top_2 and 18.73 for desk_top_1, which is much higher than 15.63 in paper) .

I also download your evaluation models here which is also about psnr 25 for the desk_top_2 scene.

There must be something wrong in my test script or the config. Could you tell me the way to repeat the result in Table 1 in ICCV21 paper? Looking forward to you reply!

Hi !

how to define the object activation codes for the object as per the number of objects in the scene ?
Say if I have only 2 objects in the scene what should be the activation library or say may be I have only 1 object what should be the activation library ?
So basically how are the activation codes assigned to the individual objects in the scene ?

In the code I saw that you use torch.embedding and input indices 1 to 5 , so does this mean the code library gives 5 activation codes for the 5 objects (cube,dragon, face, candy, magic cube).......... but which code belongs to which object ?
If I only wanted to render 3 objects out of the 5 objects then what should be done ?

Please help me with this concept of selecting the object codes @ybbbbt

Dear authors,

Thanks for your impressive work and released dataset.

I am trying to use the toydesk_data. But I wanna have the camera's intrinsic parameters.

So, could you please share it with me? Or tell me how to compute them!

Thank you very much!

Hello, thanks for your great work. It does push this research area forward a lot.
But I wonder when you guys will share your code? Can't wait to see it.