Comments (6)
AuthorityWang
commented on July 17, 2024
1
Hi @bpezet , seemingly that there is a problem with your tinycudann extension as the log shows that
OSError: Could not find compatible tinycudann extension for compute capability 61.
which suggests that this may be a problem with the module. There are similar issues in the tinycudann repo:
https://github.com/NVlabs/tiny-cuda-nn/issues?q=OSError%3A+Could+not+find+compatible+tinycudann+extension, and another issue same with you: nerfstudio-project/nerfstudio#1166. Possibly due to incompatible version of Nvidia driver or cuda toolkit. Hope this can be helpful.
from neuralangelo.
bpezet
commented on July 17, 2024
Thank's @AuthorityWang for your reply !
I followed NVlabs/tiny-cuda-nn Compilation by modifying CMakeLists.txt:118 with set(${OUT_VARIABLE} "61" PARENT_SCOPE)
Now tinycudann error disapear to let torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
# EXPERIMENT=toy_example
# GROUP=example_group
# NAME=example_name
# CONFIG=projects/neuralangelo/configs/custom/${EXPERIMENT}.yaml
# GPUS=1
# torchrun --nproc_per_node=${GPUS} train.py --logdir=logs/${GROUP}/${NAME} --config=${CONFIG} --show_pbar
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -11) local_rank: 0 (pid: 2815) of binary: /usr/bin/python
Traceback (most recent call last):
File "/usr/local/bin/torchrun", line 33, in <module>
sys.exit(load_entry_point('torch==2.1.0a0+fe05266', 'console_scripts', 'torchrun')())
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
======================================================
train.py FAILED
------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2023-08-16_12:39:34
host : 0014c90f7923
rank : 0 (local_rank: 0)
exitcode : -11 (pid: 2815)
error_file: <N/A>
traceback : Signal 11 (SIGSEGV) received by PID 2815
======================================================from neuralangelo.
AuthorityWang
commented on July 17, 2024
Hi @bpezet , I wonder if this is the problem with the PyTorch version. May I know the Torch version?
Which can be get by
python
>>>import torch
>>>print(torch.__version__)
I guess the problem is that the torch you installed is not CUDA version.
from neuralangelo.
chenhsuanlin
commented on July 17, 2024
Hi @bpezet, you may also want to set CUDA_VISIBLE_DEVICES to mask out the lower-end GPUs you may have (e.g. for powering your display). tiny-cuda-nn in our prebuilt Docker images support compute compatibility >= 70.
from neuralangelo.
bpezet
commented on July 17, 2024
Hi @AuthorityWang, I'm using prebuilt Docker images given in the repo
The PyTorch version is 2.1.0a0+fe05266
from neuralangelo.
bpezet
commented on July 17, 2024
Hi @chenhsuanlin ! Indeed, thanks to https://developer.nvidia.com/cuda-gpus I know that my GPU compatibility is < 70
I'll try again on cloud service with another GPU
Thank's to all of you for your responsiveness !
from neuralangelo.
Related Issues (20)
- Fixed poses for Courthouse no longer avaliable HOT 1
- MacBook Pro M1 compatibility HOT 1
- Colab Demo Broken HOT 1
- What is the difference between numerical taps 4 and 6？ HOT 1
- oom problem during train
- ModuleNotFoundError: No module named 'tinycudann'
- I'm curious about how the F1 score was calculated in the paper.
- CondaEnvException: Pip failed------failed to build tinycudann HOT 2
- How to add texture HOT 1
- How to "run neuralangelo"
- Save rendered images from the trained model
- Can I ask if this code can be used for scene reconstruction?
- Load a trained model and generate novel view
- how to get the envirement mesh
- Correlation of training speed influencing hyperparameters
- RGB View Synthesis
- merits
- Configuration file for Blender dataset
- Colab Notebook Throws Error: No such file or directory: 'datasets/lego_ds2/images HOT 1
- PSNR Evaluation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from neuralangelo.