gap-lab-cuhk-sz / instpifu Goto Github PK

Hi ! I have a great regard for your excellent work!
Recently, I'm trying to get the test result with your code on 3D-FUTURE. I've noticed that you have also tested on 3D-FUTURE dataset in your paper. So I wonder how to get the processed data like those in prepare_data you offered. Did you also cantact 3D-FRONT team to render the 3D-FUTURE dataset? If I try to contact them,except the dataset , what else should I prepare in advance?
Thanks a lot !!!

Hi ! I have a great regard for your excellent work!
I found the prepared SUNRGBD data download link in Total3D page is broken.
And it is a bit troublesome to process the data by ourselves according to the generation method in Total3D.
Could you please provide other download link for prepared SUNRGBD data ?
Thanks a lot !!!

Hi ! I have great regard for your excellent work!
When I try to run the sunrgbd demo, I didn't find ./data/sunrgbd/test_objects.json
Could you please upload it ?
Thanks a lot !

Hi, I just saw your updates with the added demo mode, and have been trying to run the command you included in the write up. However, I keep getting the error this error. Please let me know if I am doing anything wrong. I've tried it with the name of the render task without the file extension and with it, both giving the same error.

Hi, great work! I have some questions regarding the downsample in Dataset.

I mentioned that in 1 and 2, it says the image is downsampled by 2. However, I didn't find any downsample on image in the code (3 is commented). I'm confused. Therefore, it would be greatly appreciated if you could figure it out for me.

Thank you!

Hello guys,

Excellent work!

I am working on a project that is similar to yours and Total3D.
In the prepare data folder, you provide the dict_keys(['centroid_reg', 'coeffs_reg', 'bdb3D', 'pitch', 'roll']) for the layouts.
Could you also provide the layout centroid average and layout coefficients average over the whole data? These are equivalent to layout_avg_dict['layout_centroid_avg'] and layout_avg_dict['layout_coeffs_avg'] in Total3D implementation.

Also, I was wondering if there is a way to generate ground truth for object orientation? I want to do something similar to Total3D work and extract 'ori_cls' and 'ori_reg' parameters of each sample's layout.

Thank you! :D

In the evaluation code, only first 2000 samples are used for evaluation, but the provided gt mesh for test set doesn't include all gt meshes for those 2000 test samples. Could you release all the GT meshes for test set?

May I ask how did you generate the watertight meshes? I used ManifoldPlus , but the generated mesh is very big.

Hi Guys,

Amazing repository !
very impressed with your work.

Do you stil plan to release to code on background reconstruction and 3D Object detection ?

Regards,

I am trying to run object detection with InstPIFu, so I ran the command below:

python3 main.py --mode train --config ./configs/inference_object_detection.yaml

The following error appears:

Loading optimizer.
Traceback (most recent call last):
  File "main.py", line 25, in <module>
    train.run(cfg)
  File "/amydata/InstPIFu/train.py", line 27, in run
    optimizer = get_optimizer(config=cfg.config, net=net)
  File "/amydata/InstPIFu/net_utils/train_test_utils.py", line 275, in get_optimizer
    if config['optimizer']['type'] == 'AdamW':

I copied optimizer and scheduler configs from instpifu_train.yaml into inference_object_detection.yaml and tried running the command again. Then, the following error appears:

Traceback (most recent call last):
  File "main.py", line 25, in <module>
    train.run(cfg)
  File "/amydata/InstPIFu/train.py", line 40, in run
    trainer = get_trainer(cfg.config)
  File "/amydata/InstPIFu/net_utils/train_test_utils.py", line 379, in get_trainer
    raise NotImplementedError
NotImplementedError

Here is where the error is thrown (net_utils/train_test_utils.py):

def get_trainer(config):
    if config["method"]=="instPIFu" or config['method']=="bgPIFu":
        from training import Recon_trainer
        trainer=Recon_trainer
    else:
        raise NotImplementedError
    return trainer

In training.py, there does not appear to by any detection trainer to import.

Thank you for providing good work!

When inferring with 3D-FRONT data, results are only at the object level.
As in the paper, how can I get the scene reconstruction result considering the spatial relationship between each object in the scene?

Additionally, I wonder why occ is needed. I think the occ serves as a template, so doesn't that eliminate the need for PIFu here?

Hello, thank you very much for your excellent work!
When I load the pretrained weight of the background(model_best_bg.pth), the following problem occurred. Can you update the pretrained weights for background reconstruction again?

*** RuntimeError: Error(s) in loading state_dict for BGPIFu_Net:
        Missing key(s) in state_dict: "global_encoder.conv1.weight", "global_encoder.bn1.weight", "global_encoder.bn1.bias", "global_encoder.bn1.running_mean", "global_encoder.bn1.running_var", "global_encoder.layer1.0.conv1.weight", "global_encoder.layer1.0.conv2.weight", "global_encoder.layer1.1.conv1.weight", "global_encoder.layer1.1.conv2.weight", "global_encoder.layer2.0.conv1.weight", "global_encoder.layer2.0.conv2.weight", "global_encoder.layer2.0.downsample.0.weight", "global_encoder.layer2.0.downsample.1.weight", "global_encoder.layer2.0.downsample.1.bias", "global_encoder.layer2.0.downsample.1.running_mean", "global_encoder.layer2.0.downsample.1.running_var", "global_encoder.layer2.1.conv1.weight", "global_encoder.layer2.1.conv2.weight", "global_encoder.layer3.0.conv1.weight", "global_encoder.layer3.0.conv2.weight", "global_encoder.layer3.0.downsample.0.weight", "global_encoder.layer3.0.downsample.1.weight", "global_encoder.layer3.0.downsample.1.bias", "global_encoder.layer3.0.downsample.1.running_mean", "global_encoder.layer3.0.downsample.1.running_var", "global_encoder.layer3.1.conv1.weight", "global_encoder.layer3.1.conv2.weight", "global_encoder.layer4.0.conv1.weight", "global_encoder.layer4.0.conv2.weight", "global_encoder.layer4.0.downsample.0.weight", "global_encoder.layer4.0.downsample.1.weight", "global_encoder.layer4.0.downsample.1.bias", "global_encoder.layer4.0.downsample.1.running_mean", "global_encoder.layer4.0.downsample.1.running_var", "global_encoder.layer4.1.conv1.weight", "global_encoder.layer4.1.conv2.weight", "global_encoder.fc.weight", "global_encoder.fc.bias".
        Unexpected key(s) in state_dict: "global_surface_classifier.conv0.weight", "global_surface_classifier.conv0.bias", "global_surface_classifier.conv1.weight", "global_surface_classifier.conv1.bias", "global_surface_classifier.conv2.weight", "global_surface_classifier.conv2.bias", "global_surface_classifier.conv3.weight", "global_surface_classifier.conv3.bias", "mask_decoder.0.weight", "mask_decoder.0.bias", "mask_decoder.2.weight", "mask_decoder.2.bias", "mask_decoder.4.weight", "mask_decoder.4.bias", "mask_decoder.6.weight", "mask_decoder.6.bias", "post_op_module.channel_mlp.0.weight", "post_op_module.channel_mlp.0.bias", "post_op_module.channel_mlp.2.weight", "post_op_module.channel_mlp.2.bias", "post_op_module.channel_mlp.4.weight", "post_op_module.channel_mlp.4.bias", "post_op_module.post_conv.0.weight", "post_op_module.post_conv.0.bias", "post_op_module.post_conv.2.weight", "post_op_module.post_conv.2.bias", "post_op_module.post_conv.4.weight", "post_op_module.post_conv.4.bias", "post_op_module.post_conv.6.weight", "post_op_module.post_conv.6.bias", "post_op_module.pre_conv.0.weight", "post_op_module.pre_conv.0.bias", "post_op_module.pre_conv.2.weight", "post_op_module.pre_conv.2.bias", "global_encoder.0.weight", "global_encoder.0.bias", "global_encoder.1.conv1.weight", "global_encoder.1.bn1.weight", "global_encoder.1.bn1.bias", "global_encoder.1.bn1.running_mean", "global_encoder.1.bn1.running_var", "global_encoder.1.bn1.num_batches_tracked", "global_encoder.1.layer1.0.conv1.weight", "global_encoder.1.layer1.0.conv2.weight", "global_encoder.1.layer1.1.conv1.weight", "global_encoder.1.layer1.1.conv2.weight", "global_encoder.1.layer2.0.conv1.weight", "global_encoder.1.layer2.0.conv2.weight", "global_encoder.1.layer2.0.downsample.0.weight", "global_encoder.1.layer2.0.downsample.1.weight", "global_encoder.1.layer2.0.downsample.1.bias", "global_encoder.1.layer2.0.downsample.1.running_mean", "global_encoder.1.layer2.0.downsample.1.running_var", "global_encoder.1.layer2.0.downsample.1.num_batches_tracked", "global_encoder.1.layer2.1.conv1.weight", "global_encoder.1.layer2.1.conv2.weight", "global_encoder.1.layer3.0.conv1.weight", "global_encoder.1.layer3.0.conv2.weight", "global_encoder.1.layer3.0.downsample.0.weight", "global_encoder.1.layer3.0.downsample.1.weight", "global_encoder.1.layer3.0.downsample.1.bias", "global_encoder.1.layer3.0.downsample.1.running_mean", "global_encoder.1.layer3.0.downsample.1.running_var", "global_encoder.1.layer3.0.downsample.1.num_batches_tracked", "global_encoder.1.layer3.1.conv1.weight", "global_encoder.1.layer3.1.conv2.weight", "global_encoder.1.layer4.0.conv1.weight", "global_encoder.1.layer4.0.conv2.weight", "global_encoder.1.layer4.0.downsample.0.weight", "global_encoder.1.layer4.0.downsample.1.weight", "global_encoder.1.layer4.0.downsample.1.bias", "global_encoder.1.layer4.0.downsample.1.running_mean", "global_encoder.1.layer4.0.downsample.1.running_var", "global_encoder.1.layer4.0.downsample.1.num_batches_tracked", "global_encoder.1.layer4.1.conv1.weight", "global_encoder.1.layer4.1.conv2.weight", "global_encoder.2.weight", "global_encoder.2.bias", "global_encoder.4.weight", "global_encoder.4.bias", "global_encoder.6.weight", "global_encoder.6.bias".
        size mismatch for surface_classifier.conv0.weight: copying a param with shape torch.Size([1024, 549, 1]) from checkpoint, the shape in current model is torch.Size([1024, 1283, 1]).
        size mismatch for surface_classifier.conv1.weight: copying a param with shape torch.Size([512, 1573, 1]) from checkpoint, the shape in current model is torch.Size([512, 2307, 1]).
        size mismatch for surface_classifier.conv2.weight: copying a param with shape torch.Size([256, 1061, 1]) from checkpoint, the shape in current model is torch.Size([256, 1795, 1]).
        size mismatch for surface_classifier.conv3.weight: copying a param with shape torch.Size([128, 805, 1]) from checkpoint, the shape in current model is torch.Size([128, 1539, 1]).
        size mismatch for surface_classifier.conv4.weight: copying a param with shape torch.Size([1, 677, 1]) from checkpoint, the shape in current model is torch.Size([1, 1411, 1]).

您好,您的代码中有生成Demo的程序么?

Thanks for your sharing.
I try to train the background reconstruction model you posted recently. But I find that the data written in bg_split does not exist in the folder prepare_data.
I want to know the prepare_data and bg_occ are needed for background reconstruction.

Thanks!

Hello author, I reproduced the paper with the latest code provided by you, and got the following results:

This is quite different from the results in the paper. In the end, I used the pre-trained model provided by you and the latest code provided by you for testing and CD evaluation. The results obtained are similar to the CD distance I reproduced myself, as follows:

May I ask if the code is not perfect or is there something wrong with my reproduction process?

Hi @UncleMEDM,

Nice work! I wonder if you can release the code for rendering the 3D-Front dataset.

Thank you

Hi, thanks for your awesome work! I faced up to some problems when I follow your instruction to try the test code.
After finish downloading the prepare_data.zip , I fail to extract the archive file:

Is there any problem with the zip file? I have downloaded the data again, but the same error is occurring.
@UncleMEDM Could you help me figure out what's problem it could be?

Hi ! I have a great regard for your excellent work!
Recently, I want to use the pix3d dataset to reproduce the experimental results, and I want to visualize the reconstructed scene. I found that there are no these two parts in the code that has been released. Can you open the code of these two parts? I am in a hurry to use it recently. Or when can this part of the code be opened. Thanks for your work!

Hello, I was hoping that the code for the SUN RGBD loader for InstPIFu can be uploaded. The paper had mentioned SUN RGBD was used to compare with Im3D but the code is not yet released. Can the SUN RGBD loader and evaluation scripts for it be released please?
Thanks

Thank you for sharing your code.

I want to draw the projected ground truth 3D bounding boxes onto the 2d image, however, I couldn't extract the corners of the 3d boxes in 2d correctly. Do you have an idea how can I do it?

Thanks!

Hi, I'm currently testing out the object detection module in this repo and I ran into key error.

As written in README, I ran the code with this command:

python main.py --mode train --config ./configs/inference_object_detection.yaml

And this error came up:

File ".../net_utils/train_test_utils.py", line 275, in get_optimizer
    if config['optimizer']['type'] == 'AdamW':
KeyError: 'optimizer'

I looked into other config files, and I can't find the optimizer configurations.
Would you be so kind to look into this?

Thank you very much!

Hello, I want to use the 'depth_map' data but I don't know how to scale them into the real one. Could you please release the scale method?

Hi, nice work!

I'm confused that gt test mesh only has 1038 samples, while the test set actually has 2000 samples.
Could you please double check this?

Thank you!

Thanks for your sharing.
You mention in your paper that you have also trained and tested the network with the pix3d data set and get good results. However, when I tested with pix3d data set, I found that pix3d data set lacks two attributes: data_dict["bbox_size"] and data_dict["obj_cam_center"] , compared with 3dfrond data set.
May I ask how did you obtain these two attributes to complete the test? Or could you show how you test on pix3d?

Thanks!

Hi, from the training_data in microsoft sharpoint linked in readme, my group and I had downloaded bgocc.zip file. However we encounter issues with it when we try and unzip it in a server, and it actually failed to download locally (download got automatically cancelled). I'm attaching a screenshot of our issue when unzip it. We assume that the file might be corrupted.

Hello, and thank you for your great work!

I read your paper and saw that your main model is basically trained on the 3D-Front dataset and then fine-tuned on the Pix3D dataset for evaluation on the SUNRGBD.

My question is about the model weight file you shared, named 'model_best_pix3d.pth'. Is this model trained only on the Pix3D dataset? Or was it initially trained on 3D-Front and then fine-tuned on Pix3D?

Hi! First of all, great work! The first paper after Im3D that finally uses the whole 2D object, and not just the global feature, to reconstruct the 3D objects!

Unfortunately, while running the demo as described in the 'README.md', I encounter an issue where the reconstructed objects are oriented incorrectly. Below you can find the list of steps to reproduce.

First, fix 'demo.py', as the hardcoded 'test_all' file doesn't exist:

     instPIFu_config=CONFIG(instPIFu_config_path).config
     bg_config=CONFIG(bg_config_path).config
-    instPIFu_config['data']['test_class_name']="test_all"
+    instPIFu_config['data']['test_class_name']="all"
     instPIFu_config['data']['use_pred_pose']=True #to use predict pose or not
     instPIFu_model=InstPIFu(instPIFu_config).cuda()
     instPIFu_checkpoints=torch.load(instPIFu_config["weight"])
     instPIFu_net_weight=instPIFu_checkpoints['net']

Then run:

    python demo.py --testid rendertask6452

The results look like this:

Apart from the orientation, some objects are missing, as they are not listed in the 'all.json' file.