zju3dv / autorecon Goto Github PK

Thanks for releasing this great work!

After following your (updated) installation guide and downloading the demo data, I tried running the demo 'exps/code-release/run_pipeline_demo_low-res.sh'.

After a few initial steps it crashes with
2023-07-14 17:16:04.621 | WARNING | auto_decomp.sfm.sfm:main:632 | - [custom_data_example/co3d_chair] Unknown error: [Errno 2] No such file or directory: 'colmap'

File "/home/username/workspace/external/3drecontruction_stuff/AutoRecon/third_party/AutoDecomp/auto_decomp/sfm/sfm.py", line 173, in run
raise err
File "/home/username/workspace/external/3drecontruction_stuff/AutoRecon/third_party/AutoDecomp/auto_decomp/sfm/sfm.py", line 170, in run
self._run(task_inst_id)
File "/home/username/workspace/external/3drecontruction_stuff/AutoRecon/third_party/AutoDecomp/auto_decomp/sfm/sfm.py", line 192, in _run
self.manhattan_alignment(task)
File "/home/username/workspace/external/3drecontruction_stuff/AutoRecon/third_party/AutoDecomp/auto_decomp/sfm/sfm.py", line 247, in manhattan_alignment
rec = manhattan_alignment(task.image_dir, task.sfm_dir, task.manhattan_sfm_dir)
File "/home/username/workspace/external/3drecontruction_stuff/AutoRecon/third_party/AutoDecomp/auto_decomp/sfm/sfm.py", line 416, in manhattan_alignment
subprocess.run(cmd, check=True, capture_output=True, text=True)
File "/home/username/anaconda3/envs/auto_recon/lib/python3.9/subprocess.py", line 505, in run
with Popen(*popenargs, **kwargs) as process:
File "/home/username/anaconda3/envs/auto_recon/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/home/username/anaconda3/envs/auto_recon/lib/python3.9/subprocess.py", line 1837, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'colmap'

Having read the paper and realising how out-of-the-box the ordeal for your approach is, I'm really excited and curious for the release.

When are you planning the code-release?

Hello! Many thanks for this great project!

When I run the scripts/run_pipeline_demo_low-res.sh，a terrible error was encountered, as


my run_pipeline_demo_low-res.sh is as the following:

can you help me?

Hi

First of all, thank you for sharing this fantastic project!

It seems on the project page and in the paper that the input can be as little as 9 images in the wild, but it uses around 200 images in a sequential capture when running the project on custom data. Since I've played with hloc in the past, I know that it can't use a low amount of sparse imagery so I wonder how you were able to do a 3D asset from just 9 images in the project page.

I was replicating your project and initially installed it with CUDA 12.3. Following your instructions, I set up the environment with Python 3.10 since the 'tiny-cuda' package wasn't available. The environment creation steps are as follows:

conda create --name auto_recon -y python=3.10
conda activate auto_recon
python -m pip install --upgrade pip setuptools

Following your recommendations, I proceeded with the installation of AutoDecomp and I installed PyTorch using the following commands:

conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia

With these commands, I created an environment containing a different version of CUDA (11.7) compared to the system's CUDA (12.3).
The installed version of PyTorch3D obtained from Anaconda Cloud using the command:

conda install pytorch3d -c pytorch3d

is:

pytorch3d 0.7.5 py310_cu117_pyt1131 pytorch3d

For the 'tiny-cuda' installation, I initially attempted the following command:

pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

However, the system reported an incompatibility between the CUDA version of 'tiny-cuda' and the system's CUDA version (12.3).

Upon further investigation on this issue at this link:

NVlabs/tiny-cuda-nn#372

I discovered that it's possible to install 'tiny-cuda' with Python 3.10 and CUDA 11.7 (my Anaconda environment). I used the following command for installation:

pip install https://huggingface.co/camenduru/CoDeF/resolve/main/tinycudann-1.7-cp310-cp310-linux_x86_64.whl

This successfully installed 'tiny-cuda', after which I followed all other instructions to complete the project.
However, upon running the command:

bash exps/code-release/run_pipeline_demo_low-res.sh

I encountered an error related to 'tinycuda_bindings':

Error executing job with overrides: ['data_root=data', 'inst_rel_dir=custom_data_example/co3d_chair', 'sparse_recon.n_images=40', 'sparse_recon.force_rerun=True', 'sparse_recon.n_feature_workers=1', 'sparse_recon.n_recon_workers=1', 'triangulation.force_rerun=True', 'triangulation.n_feature_workers=1', 'triangulation.n_recon_workers=1', 'dino_feature.force_extract=True', 'dino_feature.n_workers=1']
Traceback (most recent call last):
File "/home/vicari/AutoRecon/third_party/AutoDecomp/auto_decomp/cli/inference_transformer.py", line 94, in main
data: Dict[str, Any] = sfm_preprocess()
File "/home/vicari/AutoRecon/third_party/AutoDecomp/auto_decomp/decomp/preprocess.py", line 288, in call
return SfMPreprocessWorker(self.cfg, self.cfg.inst_rel_dir, device="cuda")()
File "/home/vicari/AutoRecon/third_party/AutoDecomp/auto_decomp/decomp/preprocess.py", line 188, in call
self.parse_poses(data)
File "/home/vicari/AutoRecon/third_party/AutoDecomp/auto_decomp/decomp/preprocess.py", line 206, in parse_poses
data.update({"poses": colmap_utils.parse_poses(self.rec)})
File "/home/vicari/AutoRecon/third_party/AutoDecomp/auto_decomp/utils/colmap.py", line 23, in parse_poses
all_T44_w2c = read_extrinsics(rec)
File "/home/vicari/AutoRecon/third_party/AutoDecomp/auto_decomp/utils/colmap.py", line 11, in read_extrinsics
T44_w2c = np.concatenate([np.concatenate([img.rotmat(), img.tvec[:, None]], 1), bottom], 0)
AttributeError: 'pycolmap.Image' object has no attribute 'rotmat'
(pid=3830560) Extracting LoFTR matches: 0: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 410/410 [06:16<00:00, 1.74it/s]
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Traceback (most recent call last):
File "/home/shared/anaconda3/envs/auto_recon/bin/ns-train", line 5, in
from scripts.train import entrypoint
File "/home/vicari/AutoRecon/scripts/train.py", line 50, in
from nerfstudio.configs.method_configs import AnnotatedBaseConfigUnion
File "/home/vicari/AutoRecon/nerfstudio/configs/method_configs.py", line 26, in
from nerfstudio.configs.autorecon import method_configs as autorecon_method_configs
File "/home/vicari/AutoRecon/nerfstudio/configs/autorecon/init.py", line 44, in
from nerfstudio.fields.sdf_field import SDFFieldConfig
File "/home/vicari/AutoRecon/nerfstudio/fields/sdf_field.py", line 42, in
from nerfstudio.field_components.progressive_encoding import ProgressiveEncoding
File "/home/vicari/AutoRecon/nerfstudio/field_components/progressive_encoding.py", line 5, in
import tinycudann as tcnn
File "/home/shared/anaconda3/envs/auto_recon/lib/python3.10/site-packages/tinycudann/init.py", line 9, in
from tinycudann.modules import free_temporary_memory, NetworkWithInputEncoding, Network, Encoding
File "/home/shared/anaconda3/envs/auto_recon/lib/python3.10/site-packages/tinycudann/modules.py", line 51, in
C = importlib.import_module(f"tinycudann_bindings.{cc}_C")
File "/home/shared/anaconda3/envs/auto_recon/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: /home/shared/anaconda3/envs/auto_recon/lib/python3.10/site-packages/tinycudann_bindings/_75_C.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda20CUDACachingAllocator9allocatorE

Could you provide insights into resolving this error?

When I provide the training script with sparse sequential images (approx 20 for the scene), the "Computing & saving fg bbox masks" step taking a long time to finish and it doesn't get through 1 or 2 iterations before using up all the computer's 32GB RAM space. How can I provide the training script with sparse images to do the task?

TypeError: 'NoneType' object is not subscriptable, when export texture mesh
When i export the texture mesh use the following command :
`LOG_DIR="outputs/neusfacto-wbg-reg_sep-plane-nerf_60k_plane-h-ratio-0.3_engine_337/neus-facto-wbg-reg_sep-plane-nerf/2023-07-20_033253"
MC_RES=512
MESH_FN="extracted_texture_mesh_res-${MC_RES}.ply"
MESH_PATH="${LOG_DIR}/${MESH_FN}"

ns-extract-mesh
--load-config $LOG_DIR/config.yml
--load-dir $LOG_DIR/sdfstudio_models
--output-path $MESH_PATH
--chunk_size 25000 --store_float16 True
--resolution $MC_RES
--use_train_scene_box True
--seg_aware_sdf False
--remove_internal_geometry None
--remove_non_maximum_connected_components False
--close_holes False --simplify_mesh_final False
--extract_texture True`

This error always occured:

Could u give me some advice to solve this problem?
Thank u very much!!!

You mention this in the documentation:
NOTE: In the demo script, we assume the images come from a sequential video stream and use sequential matching for SfM. If you have unordered images, the default setting might lead to inferior result and it is recommended to use exhaustive matching or vocab-tree instead.

Where should we make the change to switch from sequential to exhaustive matching?

Traceback (most recent call last):
File "/home/ljj/AutoRecon-main/third_party/AutoDecomp/auto_decomp/inference_transformer.py", line 79, in main
sfm.main(cfg.sparse_recon)
File "/home/ljj/AutoDecomp/auto_decomp/sfm/sfm.py", line 610, in main
reconstruct_instance(copy.deepcopy(args), args.inst_rel_dir, *_queue_args, task_id=0)
File "/home/ljj/AutoDecomp/auto_decomp/sfm/sfm.py", line 521, in reconstruct_instance
pairs_from_sequential.main(
File "/home/ljj/AutoDecomp/auto_decomp/sfm/pairs_from_sequential.py", line 129, in main
loop_det_pairs = build_loop_detection_pairs(
File "/home/ljj/AutoDecomp/auto_decomp/sfm/pairs_from_sequential.py", line 61, in build_loop_detection_pairs
retrival_feat_path = extract_features.main(
File "/root/miniconda3/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/root/3Dvision/AutoRecon-main/third_party/AutoDecomp/third_party/Hierarchical-Localization/hloc/extract_features.py", line 253, in main
model = Model(conf['model']).eval().to(device)
File "/root/3Dvision/AutoRecon-main/third_party/AutoDecomp/third_party/Hierarchical-Localization/hloc/utils/base_model.py", line 17, in init
self._init(conf)
File "/root/3Dvision/AutoRecon-main/third_party/AutoDecomp/third_party/Hierarchical-Localization/hloc/extractors/netvlad.py", line 86, in _init
mat['net'].layers):
AttributeError: 'numpy.ndarray' object has no attribute 'layers'

Hello, thanks for this interesting work!

I would like to ask whether it is somehow integrated within the nerfstudio folder or elsewhere

Thanks in advance,

Alberto

[Errno 2] No such file or directory: 'data/CO3D_DEMO/scan1/triangulate_loftr-720000_sequential_np-10/auto-deocomp_sfm-tra nsformer_cvpr/cameras_cameras_norm-obj-side-2.0.npz'

Could you please tell me how to fix it?

Describe the bug
A clear and concise description of what the bug is.

https://colab.research.google.com/github/nerfstudio-project/nerfstudio/blob/tancik%2Fpolycam/colab/demo.ipynb

could you please provide a minimal interactive google colab where I can input my own images in order to extract the mesh?
Thanks a lot for your great contribution to open-source community

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

This is a commendable job！

Can you provide a demo on how to apply textures and generate Object Foreground Masks？

Amazing project! Is it possible to run the code on the RTX3060 12GB VRAM?

When cloning via
git clone --recurse-submodules [email protected]:zju3dv/AutoRecon.git

Submodule path 'third_party/AutoDecomp/third_party/Hierarchical-Localization/third_party/disk/submodules/torch-dimcheck': checked out '1b141ec38101500c59ae11ddb628e2fa3e40f18e'
Submodule path 'third_party/AutoDecomp/third_party/Hierarchical-Localization/third_party/disk/submodules/torch-localize': checked out 'f6dc5818ece911b6fed5bbdc3ce8aa2e60322116'
Submodule path 'third_party/AutoDecomp/third_party/Hierarchical-Localization/third_party/disk/submodules/unets': checked out 'af4d574c13b3f137115394e714522b815fc6a417'
Submodule path 'third_party/AutoDecomp/third_party/Hierarchical-Localization/third_party/r2d2': checked out 'd6777a9d6769448998e5abe11031ae05de28e49a'
Submodule path 'third_party/AutoDecomp/third_party/LoFTR': checked out '324dedc4ea1b36a5fc57b08d428a0f7548fdc06f'
Submodule 'third_party/SuperGluePretrainedNetwork' ([email protected]:magicleap/SuperGluePretrainedNetwork.git) registered for path 'third_party/AutoDecomp/third_party/LoFTR/third_party/SuperGluePretrainedNetwork'
Cloning into '/home/pablo/0Dev/repos/AutoRecon/third_party/AutoDecomp/third_party/LoFTR/third_party/SuperGluePretrainedNetwork'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:magicleap/SuperGluePretrainedNetwork.git' into submodule path '/home/pablo/0Dev/repos/AutoRecon/third_party/AutoDecomp/third_party/LoFTR/third_party/SuperGluePretrainedNetwork' failed
Failed to clone 'third_party/SuperGluePretrainedNetwork'. Retry scheduled
Cloning into '/home/pablo/0Dev/repos/AutoRecon/third_party/AutoDecomp/third_party/LoFTR/third_party/SuperGluePretrainedNetwork'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:magicleap/SuperGluePretrainedNetwork.git' into submodule path '/home/pablo/0Dev/repos/AutoRecon/third_party/AutoDecomp/third_party/LoFTR/third_party/SuperGluePretrainedNetwork' failed
Failed to clone 'third_party/SuperGluePretrainedNetwork' a second time, aborting
fatal: Failed to recurse into submodule path 'third_party/AutoDecomp/third_party/LoFTR'
fatal: Failed to recurse into submodule path 'third_party/AutoDecomp'

Describe the bug
I just set this project up, but the installation instructions were incomplete.
I'm not sure if this should go in the AutoRecon or the AutoDecomp docs, so I figured I'd make an issue rather than a PR.
There are 2 steps missing from the installation instructions to get this working: COLMAP and the LoFTR pretrained models.

COLMAP can, if it isn't already be present, be installed through conda install -y -c conda-forge colmap, but the package doesn't always play very nice with conda, so it may be preferred to do conda install -y -c conda-forge mamba && mamba install -y -c conda-forge colmap, which can save literally hours of time sometimes.

As for LoFTR, they provide a page to download the pretrained models that this project uses in the installation section: https://github.com/zju3dv/LoFTR#installation. The weights (or at least the outdoor weights this project uses) need to be extracted to AutoRecon/third_party/AutoDecomp/third_party/LoFTR/weights/ manually.

After this, I can modify the low_res_demo script and successfully run it on my own data.

Sidenote: Have you considered upstreaming your work into Nerfstudio properly? There are more methods in there that have external dependencies, and getting your project upstreamed would let it make use of all the many improvements in Nerfstudio upstream, while making it easier to stay up-to-date.

Hi,

thanks for your great project! I've just finished to read the paper. May I know what software/library did you use for SfM? I see you add a few modifications to the standard SfM pipeline (e.g. LoFTR matcher instead sparse key points). just curious what open-source SfM library is easy to extend and best suitable for this task