facebookresearch / habitat-challenge Goto Github PK
View Code? Open in Web Editor NEWCode for the habitat challenge
Home Page: https://aihabitat.org
License: MIT License
Code for the habitat challenge
Home Page: https://aihabitat.org
License: MIT License
Hi,
Few clarification questions about runtime limits for the pointnav task
What is the time limit for test-standard and test-challenge phase? The README mentions 24 hours, but i have noticed in the logs "Timelimit for test-std, test-challenge track is 151200 seconds" (42 hours)
What is the number of evaluation episodes?
Unfortunately my solution is currently rather slow, taking around 2 minutes per episode on average. 2000 episodes would take around 66 hours. Is it possible to extend the time limit?
I have also noticed that if my script does not terminate within the time limit there is no output generated and the submission is marked failed. Would it be possible to generate results, maybe treating any remaining episodes as failed? I have done something like this in my script (always take stop action if total execution time is beyond limit), but this limits my runtime to a specified amount, and I cannot be sure how much extra time will be spent on loading remaining episodes
thank you
My code requires this library for running: https://github.com/rusty1s/pytorch_scatter . When I try installing it, it throws out the following error:
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/torch_scatter
copying torch_scatter/sub.py -> build/lib.linux-x86_64-3.6/torch_scatter
copying torch_scatter/add.py -> build/lib.linux-x86_64-3.6/torch_scatter
copying torch_scatter/div.py -> build/lib.linux-x86_64-3.6/torch_scatter
copying torch_scatter/mul.py -> build/lib.linux-x86_64-3.6/torch_scatter
copying torch_scatter/std.py -> build/lib.linux-x86_64-3.6/torch_scatter
copying torch_scatter/__init__.py -> build/lib.linux-x86_64-3.6/torch_scatter
copying torch_scatter/mean.py -> build/lib.linux-x86_64-3.6/torch_scatter
copying torch_scatter/logsumexp.py -> build/lib.linux-x86_64-3.6/torch_scatter
copying torch_scatter/min.py -> build/lib.linux-x86_64-3.6/torch_scatter
copying torch_scatter/max.py -> build/lib.linux-x86_64-3.6/torch_scatter
creating build/lib.linux-x86_64-3.6/test
copying test/test_backward.py -> build/lib.linux-x86_64-3.6/test
copying test/__init__.py -> build/lib.linux-x86_64-3.6/test
copying test/test_logsumexp.py -> build/lib.linux-x86_64-3.6/test
copying test/test_max_min.py -> build/lib.linux-x86_64-3.6/test
copying test/utils.py -> build/lib.linux-x86_64-3.6/test
copying test/test_std.py -> build/lib.linux-x86_64-3.6/test
copying test/test_multi_gpu.py -> build/lib.linux-x86_64-3.6/test
copying test/test_forward.py -> build/lib.linux-x86_64-3.6/test
copying test/test_broadcasting.py -> build/lib.linux-x86_64-3.6/test
creating build/lib.linux-x86_64-3.6/torch_scatter/composite
copying torch_scatter/composite/__init__.py -> build/lib.linux-x86_64-3.6/torch_scatter/composite
copying torch_scatter/composite/softmax.py -> build/lib.linux-x86_64-3.6/torch_scatter/composite
creating build/lib.linux-x86_64-3.6/torch_scatter/utils
copying torch_scatter/utils/ext.py -> build/lib.linux-x86_64-3.6/torch_scatter/utils
copying torch_scatter/utils/__init__.py -> build/lib.linux-x86_64-3.6/torch_scatter/utils
copying torch_scatter/utils/gen.py -> build/lib.linux-x86_64-3.6/torch_scatter/utils
running build_ext
building 'torch_scatter.scatter_cpu' extension
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/cpu
gcc -pthread -B /opt/conda/envs/habitat/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/TH -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/THC -I/opt/conda/envs/habitat/include/python3.6m -c cpu/scatter.cpp -o build/temp.linux-x86_64-3.6/cpu/scatter.o -Wno-unused-variable -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=scatter_cpu -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
g++ -pthread -shared -B /opt/conda/envs/habitat/compiler_compat -L/opt/conda/envs/habitat/lib -Wl,-rpath=/opt/conda/envs/habitat/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/cpu/scatter.o -o build/lib.linux-x86_64-3.6/torch_scatter/scatter_cpu.cpython-36m-x86_64-linux-gnu.so
building 'torch_scatter.scatter_cuda' extension
creating build/temp.linux-x86_64-3.6/cuda
gcc -pthread -B /opt/conda/envs/habitat/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/TH -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/envs/habitat/include/python3.6m -c cuda/scatter.cpp -o build/temp.linux-x86_64-3.6/cuda/scatter.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=scatter_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
/usr/local/cuda/bin/nvcc -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/TH -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/envs/habitat/include/python3.6m -c cuda/scatter_kernel.cu -o build/temp.linux-x86_64-3.6/cuda/scatter_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=scatter_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
unable to execute '/usr/local/cuda/bin/nvcc': No such file or directory
error: command '/usr/local/cuda/bin/nvcc' failed with exit status 1
----------------------------------------
ERROR: Failed building wheel for torch-scatter
Looks like nvcc is missing. I tried manually looking at the base docker. I couldn't find nvcc. Should I manually install it in that case?
I submitted to ObjectNav Test-Challenge. On EvalAI, the submission turned up as Cancelled almost immediately after submitting. The submission appeared again (about 10 min later) as a second submission on the site and was Running for 229350 sec. This job ultimately ended with the status Failed with blank result, std out and std err files.
I submitted exactly the same container to the minival phase 1 time and the Test-Std phase 2 times. The minival submission completed successfully with results. The first Test-Std submission ended with a Failed status and blank result, std out and std err files. The second Test-Std submission completed successfully with results.
Hi, I have made a couple of remote submissions recently (pointnav test-std) but they all fail. I have seen two types of failure cases, both happened multiple times.
1, script fails after a few hours, and it produces an stderr output with the following error:
Traceback (most recent call last):
File "agent.py", line 172, in <module>
main()
File "agent.py", line 168, in main
challenge.submit(agent)
File "/habitat-api/habitat/core/challenge.py", line 19, in submit
metrics = super().evaluate(agent)
File "/habitat-api/habitat/core/benchmark.py", line 171, in evaluate
return self.remote_evaluate(agent, num_episodes)
File "/habitat-api/habitat/core/benchmark.py", line 103, in remote_evaluate
SerializedEntity=pack_for_grpc(action)
File "/opt/conda/envs/habitat/lib/python3.6/site-packages/grpc/_channel.py", line 826, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/opt/conda/envs/habitat/lib/python3.6/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string = "{"created":"@1599858112.185563913","description":"Error received from peer ipv4:127.0.0.1:8085","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket closed","grpc_status":14}"
2, the script fails after 42 hours and it produces empty stdout and stderr files.
For the past two days I have been unable to make new submissions to the EvalAI site. I even re-cloned the repo, cleared my docker caches and tried to resubmit the random baseline agent with no success.
Previously I was able to make submissions, so I am wondering if the issue is on the EvalAI platform?
I created a topic on their forum but I have had no response yet, if you are able to confirm that the platform is still working that would be much appreciated. (for example by submitting a random agent)
I thought it was better to raise a separate issue for this. I'm observing large inconsistencies in the local and remote submissions. I made 3 submissions of the same model on the minival track. I also evaluated the docker on a local server. These were the results I got:
SPL | Success | Distance to goal | |
---|---|---|---|
Trial 1 | 0.170 | 0.244 | 2.630 |
Trial 2 | 0.150 | 0.206 | 2.491 |
Trial 3 | 0.172 | 0.234 | 2.486 |
SPL | Success | Distance to goal | |
---|---|---|---|
Trial 1 | 0.260 | 0.366 | 0.895 |
Trial 2 | 0.258 | 0.333 | 0.846 |
Trial 3 | 0.336 | 0.433 | 0.870 |
I additionally evaluated locally on a ppo_trainer
evaluation script I wrote based in habitat-baselines. I've tried to keep things as consistent as possible between my docker submission and the ppo_trainer
script.
SPL | Success | Distance to goal | |
---|---|---|---|
Trial 1 | 0.327 | 0.455 | 1.504 |
Trial 2 | 0.350 | 0.492 | 1.503 |
Trial 3 | 0.333 | 0.455 | 1.622 |
Could these inconsistencies be due to some random seed issues? I've set them to 123 as follows:
random.seed(config.PYT_RANDOM_SEED)
np.random.seed(config.PYT_RANDOM_SEED)
torch.random.manual_seed(config.PYT_RANDOM_SEED)
if torch.cuda.is_available():
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
Is it sufficient for our final models to be under submission before the deadline, or should they have strictly finished evaluating before the deadline?
I followed the submission criteria and made a docker submission for minival. Locally, it consistently reaches a given distance to goal, but remotely, it receives a different, higher distance to goal.
(where the 6.63 is distance_to_goal
)
Is this noise? I'm surprised, as I received the same results on different machines, and also different runs of the same container (as shown in first screenshot).
I followed these instructions on ubuntu:
https://colab.research.google.com/gist/mathfac/8c9b97d7afef36e377f17d587c903ede#scrollTo=Qo_z277BAueV
But when I run the test I get:
(base) ggg:~/habitat/habitat-challenge$ bash ./test_locally_pointnav_rgbd.sh --docker-name ddppo_pointnav_submission
Neither `ifconfig` (`ifconfig -a`) nor `ip` (`ip address show`) commands are available, listing network interfaces is likely to fail
Checkpoint loaded: demo.ckpt.pth
/habitat-lab/habitat_baselines/config/default.py:214: UserWarning: NUM_PROCESSES is depricated and will be removed in a future version. Use NUM_ENVIRONMENTS instead. Overwriting NUM_ENVIRONMENTS with NUM_PROCESSES for backwards compatibility.
"NUM_PROCESSES is depricated and will be removed in a future version."
Traceback (most recent call last):
File "agent.py", line 200, in <module>
main()
File "agent.py", line 189, in main
agent = DDPPOAgent(config)
File "agent.py", line 124, in __init__
for k, v in ckpt["state_dict"].items()
File "/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/nn/modules/module.py", line 847, in load_state_dict
self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for PointNavResNetPolicy:
Missing key(s) in state_dict: "net.pointgoal_embedding.weight", "net.pointgoal_embedding.bias".
Unexpected key(s) in state_dict: "net.tgt_embeding.weight", "net.tgt_embeding.bias".
size mismatch for net.visual_encoder.compression.0.weight: copying a param with shape torch.Size([128, 1024, 3, 3]) from checkpoint, the shape in current model is torch.Size([82, 1024, 3, 3]).
size mismatch for net.visual_encoder.compression.1.weight: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([82]).
size mismatch for net.visual_encoder.compression.1.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([82]).
size mismatch for net.visual_fc.1.weight: copying a param with shape torch.Size([512, 2048]) from checkpoint, the shape in current model is torch.Size([512, 2050]).
Any suggestion?
Hi,
Our submission keeps failing on test standard with "standard_error", which I am assuming is because of time limit. The same docker images takes 14mins for remote evaluation on val_mini. Assuming test standard is 33-66 times val_mini (1000-2000 episodes vs 30 episodes), our submission should take 8-16 hrs on test standard. But the submission ran for 42 hrs and failed. Is there any way to get more information such as how many episodes were completed and left, average length of episodes, the intermediate scores when the submission timed out, etc?
Any other suggestions on how to figure out the problem would be helpful.
Thanks
Hi, I've made a submission for the ObjectNav task in the test-std phase. Currently is in "submitted" state for about 5 days. Is this due to a switch from the 2020 Challenge to the 2021 Challenge? Or are there some problems with the remote server?
Thanks,
Tommaso
Troubleshooting why I could not run and try your 2021 docker images for the habitat-challenge today and it seems like they might not have been published yet on docker hub? Or am I missing something? I'm able to docker pull fairembodied/habitat-challenge:latest but trying to build from the dockerfiles and that tag causes errors.
docker pull fairembodied/habitat-challenge:testing_2021_habitat_base_docker
Error response from daemon: manifest for fairembodied/habitat-challenge:testing_2021_habitat_base_docker not found: manifest unknown: manifest unknown
thanks.
While running the "test_locally_pointnav_rgbd.sh" script, I get the following error:
2020-03-24 10:15:01,170 Initializing dataset PointNav-v1
2020-03-24 10:15:01,171 initializing sim Sim-v0
2020-03-24 10:15:02,310 Initializing task Nav-v0
Traceback (most recent call last):
File "agent.py", line 26, in <module>
main()
File "agent.py", line 21, in main
challenge = habitat.Challenge()
File "/habitat-api/habitat/core/challenge.py", line 16, in __init__
super().__init__(config_paths)
File "/habitat-api/habitat/core/benchmark.py", line 30, in __init__
self._env = Env(config=config_env)
File "/habitat-api/habitat/core/env.py", line 102, in __init__
dataset=self._dataset,
File "/habitat-api/habitat/tasks/registration.py", line 21, in make_task
return _task(**kwargs)
File "/habitat-api/habitat/tasks/nav/nav.py", line 968, in __init__
super().__init__(config=config, sim=sim, dataset=dataset)
File "/habitat-api/habitat/core/embodied_task.py", line 241, in __init__
entities_config=config,
File "/habitat-api/habitat/core/embodied_task.py", line 269, in _init_entities
entity_type = register_func(entity_cfg.TYPE)
File "/opt/conda/envs/habitat/lib/python3.6/site-packages/yacs/config.py", line 141, in __getattr__
raise AttributeError(name)
AttributeError: TYPE
When I traced the error, I found that the task type specified in "challenge_pointnav2020.local.rgbd.yaml" file is not being read by yacs. Please let me know how I can fix this problem.
In the submission instructions it states that valid phases are: habitat21-{pointnav, objectnav}-{minival, test-std, test-ch}. However, on the eval.ai submission website, the submission phases are listed in the format: habitat-objectnav-minival-2021-802.
When trying to submit with the github instructions, I get the error:
Error: Challenge phase with slug habitat21-objectnav-minival does not exist
When trying to submit with the phases listed in the eval.ai website, I get the error:
Error: Sorry, cannot accept submissions since challenge phase is not active
Note that I can successfully make 2020 minival submissions on eval.ai.
Are the correct phases activated? Also, what is the correct phase naming convention?
Thank you!
Hi,
My submission to the evalai portal failed and it is showing just "standard_error" in the Stderr File.
Could you help in figuring out the error?
Seems this code needs to build docker, how can we run it under the official habitat-api / habitat-sim environment without the docker?
Thanks a lot!
Hi there,
I found that in current ReadMe file local evaluation step 5, if a soft link is used and the suggested docker command is run,
there would be an error message like this:
2021-04-13 21:51:15,680 Initializing dataset ObjectNav-v1
2021-04-13 21:51:15,710 initializing sim Sim-v0
WARNING: Logging before InitGoogleLogging() is written to STDERR
E0413 21:51:15.723701 17 StageAttributesManager.cpp:90] StageAttributesManager::registerObjectFinalize : Render asset template handle : habitat-challenge-data/data/scene_datasets/mp3d/x8F5xyUWy9e/x8F5xyUWy9e.glb specified in stage template with handle : habitat-challenge-data/data/scene_datasets/mp3d/x8F5xyUWy9e/x8F5xyUWy9e.glb does not correspond to any existing file or primitive render asset. Aborting.
submission.sh: line 3: 17 Segmentation fault (core dumped) python agent.py --evaluation $AGENT_EVALUATION_TYPE $@`
My fix was to change this line:
-v $(realpath habitat-challenge-data/data/scene_datasets/mp3d) \
to
-v $(realpath habitat-challenge-data/data/scene_datasets/mp3d):/habitat-challenge-data/data/scene_datasets/mp3d \
Not that familiar with Docker, so not sure whether this is a version problem or a bug.
Just want to post one solution!
Best,
Yiqing
hi, I try to submit my result for minival, the status shows "submitted", but execution time and the result are None. I guess there may be some mistakes in my docker. So I follow Readme to submit the original agent and ppo agent. The status is "submitted" and others show "None".
My question is that is it possible to submit for test or minival. If so, any suggestions for my problems.
Thanks a lot.
How is the agent path length calculated in the SPL metric?
Hi. I am trying to run the test_locally_objectnav_rgbd.sh
script with the provided RandomAgent
. I am following the Local Evaluation instruction:
objectnav_submission
image usingdocker build . --file Objectnav.Dockerfile -t objectnav_submission
tree habitat-challenge-data
habitat-challenge-data
├── data
│ ├── datasets
│ │ └── objectnav
│ │ └── mp3d
│ │ └── v1
│ │ ├── objectnav_mp3d_v1.zip
│ │ ├── train
│ │ │ ├── content
│ │ │ │ ├── 17DRP5sb8fy.json.gz
│ │ │ │ ├── 1LXtFkjw3qL.json.gz
│ │ │ │ ├── 1pXnuDYAj8r.json.gz
│ │ │ │ ├── 29hnd4uzFmX.json.gz
│ │ │ │ ├── 5LpN3gDmAk7.json.gz
│ │ │ │ ├── 5q7pvUzZiYa.json.gz
│ │ │ │ ├── 759xd9YjKW5.json.gz
│ │ │ │ ├── 7y3sRwLe3Va.json.gz
│ │ │ │ ├── 82sE5b5pLXE.json.gz
│ │ │ │ ├── 8WUmhLawc2A.json.gz
│ │ │ │ ├── aayBHfsNo7d.json.gz
│ │ │ │ ├── ac26ZMwG7aT.json.gz
│ │ │ │ ├── B6ByNegPMKs.json.gz
│ │ │ │ ├── b8cTxDM8gDG.json.gz
│ │ │ │ ├── cV4RVeZvu5T.json.gz
│ │ │ │ ├── D7G3Y4RVNrH.json.gz
│ │ │ │ ├── D7N2EKCX4Sj.json.gz
│ │ │ │ ├── dhjEzFoUFzH.json.gz
│ │ │ │ ├── E9uDoFAP3SH.json.gz
│ │ │ │ ├── e9zR4mvMWw7.json.gz
│ │ │ │ ├── EDJbREhghzL.json.gz
│ │ │ │ ├── GdvgFV5R1Z5.json.gz
│ │ │ │ ├── gZ6f7yhEvPG.json.gz
│ │ │ │ ├── HxpKQynjfin.json.gz
│ │ │ │ ├── i5noydFURQK.json.gz
│ │ │ │ ├── JeFG25nYj2p.json.gz
│ │ │ │ ├── JF19kD82Mey.json.gz
│ │ │ │ ├── jh4fc5c5qoQ.json.gz
│ │ │ │ ├── kEZ7cmS4wCh.json.gz
│ │ │ │ ├── mJXqzFtmKg4.json.gz
│ │ │ │ ├── p5wJjkQkbXX.json.gz
│ │ │ │ ├── Pm6F8kyY3z2.json.gz
│ │ │ │ ├── pRbA3pwrgk9.json.gz
│ │ │ │ ├── PuKPg4mmafe.json.gz
│ │ │ │ ├── PX4nDJXEHrG.json.gz
│ │ │ │ ├── qoiz87JEwZ2.json.gz
│ │ │ │ ├── r1Q1Z4BcV1o.json.gz
│ │ │ │ ├── r47D5H71a5s.json.gz
│ │ │ │ ├── rPc6DW4iMge.json.gz
│ │ │ │ ├── s8pcmisQ38h.json.gz
│ │ │ │ ├── S9hNv5qa7GM.json.gz
│ │ │ │ ├── sKLMLpTHeUy.json.gz
│ │ │ │ ├── sT4fr6TAbpF.json.gz
│ │ │ │ ├── ULsKaCPVFJR.json.gz
│ │ │ │ ├── uNb9QFRL6hY.json.gz
│ │ │ │ ├── ur6pFq6Qu1A.json.gz
│ │ │ │ ├── Uxmj2M2itWa.json.gz
│ │ │ │ ├── V2XKFyX4ASd.json.gz
│ │ │ │ ├── VFuaQ6m2Qom.json.gz
│ │ │ │ ├── VLzqgDo317F.json.gz
│ │ │ │ ├── VVfe2KiqLaN.json.gz
│ │ │ │ ├── Vvot9Ly1tCj.json.gz
│ │ │ │ ├── vyrNrziPKCB.json.gz
│ │ │ │ ├── XcA2TqTSSAj.json.gz
│ │ │ │ ├── YmJkqBEsHnH.json.gz
│ │ │ │ └── ZMojNkEp431.json.gz
│ │ │ └── train.json.gz
│ │ ├── val
│ │ │ ├── content
│ │ │ │ ├── 2azQ1b91cZZ.json.gz
│ │ │ │ ├── 8194nk5LbLH.json.gz
│ │ │ │ ├── EU6Fwq7SyZv.json.gz
│ │ │ │ ├── oLBMNvg9in8.json.gz
│ │ │ │ ├── pLe4wQe7qrG.json.gz
│ │ │ │ ├── QUCTc6BB5sX.json.gz
│ │ │ │ ├── TbHJrupSAjP.json.gz
│ │ │ │ ├── X7HyMhZNoso.json.gz
│ │ │ │ ├── x8F5xyUWy9e.json.gz
│ │ │ │ ├── Z6MFQCViBuw.json.gz
│ │ │ │ └── zsNo4HB9uLZ.json.gz
│ │ │ └── val.json.gz
│ │ └── val_mini
│ │ └── val_mini.json.gz
│ └── scene_datasets
│ └── mp3d
│ ├── download_mp.py
│ └── v1
│ └── scans
│ └── 17DRP5sb8fy
│ ├── cameras.zip
│ ├── matterport_camera_intrinsics.zip
│ ├── matterport_camera_poses.zip
│ ├── matterport_color_images.zip
│ └── tmp0PJfol
├── objectgoal_mp3d
│ └── val_mini
│ └── val_mini.json.gz
└── pointgoal_gibson_v2
└── val_mini
├── content
│ └── Pablo.json.gz
└── val_mini.json.gz
❯ ./test_locally_objectnav_rgbd.sh --docker-name objectnav_submission
2021-03-19 13:32:25,265 Initializing dataset ObjectNav-v1
Traceback (most recent call last):
File "agent.py", line 41, in <module>
main()
File "agent.py", line 33, in main
challenge = habitat.Challenge(eval_remote=False)
File "/habitat-lab/habitat/core/challenge.py", line 16, in __init__
super().__init__(config_paths, eval_remote=eval_remote)
File "/habitat-lab/habitat/core/benchmark.py", line 38, in __init__
self._env = Env(config=config_env)
File "/habitat-lab/habitat/core/env.py", line 79, in __init__
id_dataset=config.DATASET.TYPE, config=config.DATASET
File "/habitat-lab/habitat/datasets/registration.py", line 20, in make_dataset
return _dataset(**kwargs) # type: ignore
File "/habitat-lab/habitat/datasets/object_nav/object_nav_dataset.py", line 74, in __init__
super().__init__(config)
File "/habitat-lab/habitat/datasets/pointnav/pointnav_dataset.py", line 93, in __init__
with gzip.open(datasetfile_path, "rt") as f:
File "/opt/conda/envs/habitat/lib/python3.6/gzip.py", line 53, in open
binary_file = GzipFile(filename, gz_mode, compresslevel)
File "/opt/conda/envs/habitat/lib/python3.6/gzip.py", line 163, in __init__
fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'habitat-challenge-data/objectnav_mp3d/val_mini/val_mini.json.gz'
Thanks for your help with this!
I ran the following command for evalai push
evalai push gp_nav_rgbd:latest --phase habitat19-rgbd-minival
and got the following error:
Traceback (most recent call last):
File "/home/dchaplot/anaconda3/lib/python3.6/site-packages/urllib3/response.py", line 302, in _error_catcher
yield
File "/home/dchaplot/anaconda3/lib/python3.6/site-packages/urllib3/response.py", line 384, in read
data = self._fp.read(amt)
File "/home/dchaplot/anaconda3/lib/python3.6/http/client.py", line 449, in read
n = self.readinto(b)
File "/home/dchaplot/anaconda3/lib/python3.6/http/client.py", line 483, in readinto
return self._readinto_chunked(b)
File "/home/dchaplot/anaconda3/lib/python3.6/http/client.py", line 578, in _readinto_chunked
chunk_left = self._get_chunk_left()
File "/home/dchaplot/anaconda3/lib/python3.6/http/client.py", line 546, in _get_chunk_left
chunk_left = self._read_next_chunk_size()
File "/home/dchaplot/anaconda3/lib/python3.6/http/client.py", line 506, in _read_next_chunk_size
line = self.fp.readline(_MAXLINE + 1)
File "/home/dchaplot/anaconda3/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
My docker image is 7.67GB.
What might be the issue here?
Hi,
it looks like the Gibson dataset link in Readme is broken, it is caused by that form has accepted maximum number of responses. Can you update to this one instead: https://forms.gle/36TW9uVpjrE1Mkf9A
Thanks a lot!
Hi @abhiskk !
I made submissions for point nav minival and test-standart phases about 24 hours ago (UCULab team) and the status for those submissions is still "Running". What could cause such a long evaluation ?
Hello, thank you for creating this years challenge!
Regarding the resolution, for object-nav the config file has a resolution of WIDTH: 640, HEIGHT: 480. Will it be possible to use lower resolutions for the challenge? I believe this was available last year but wish to confirm.
Thank you.
I have a couple of questions regarding the configurations:
configs/challenge_pointnav2020_v2.local.rgbd.yaml
which has a different sensor orientation. What is this for? Should we re-train our agents with this new configuration?configs/challenge_pointnav2020.local.rgbd_test_scene.yaml
for? It is using v1
data. Should it not be v2
?How strictly will the model evaluation time be taken into consideration? I can imagine some models taking several hours rather than 30 minutes to complete 1000+ episodes.
Our instructions say:
a) PoinNav: Download Gibson scenes used for Habitat Challenge. Accept terms here and select the download corresponding to “Habitat Challenge Data for Gibson (1.5 GB)“. Place this data in: habitat-challenge/habitat-challenge-data/gibson
b) ObjectNav: Download Matterport3D scenes used for Habitat Challenge here. Place this data in: habitat-challenge/habitat-challenge-data/mp3d
But our scripts are looking for habitat-challenge/habitat-challenge-data/scene_datasets/mp3d
, as you can see from these errors:
(base) dbatra@ubuntu-bionic-1:~/habitat-challenge$ ./test_locally_objectnav_rgbd.sh --docker-name my_submission
2020-03-05 03:15:30,744 Initializing dataset ObjectNav-v1
2020-03-05 03:15:30,776 initializing sim Sim-v0
WARNING: Logging before InitGoogleLogging() is written to STDERR
E0305 03:15:31.245062 16 ResourceManager.cpp:84] Cannot load from file habitat-challenge-data/scene_datasets/mp3d/pLe4wQe7qrG/pLe4wQe7qrG.glb
E0305 03:15:31.249548 16 Simulator.cpp:106] cannot load habitat-challenge-data/scene_datasets/mp3d/pLe4wQe7qrG/pLe4wQe7qrG.glb
If I create appropriate symlinks, this error goes away:
(base) dbatra@ubuntu-bionic-1:~/habitat-challenge$ ./test_locally_objectnav_rgbd.sh --docker-name my_submission
2020-03-05 03:36:44,704 Initializing dataset ObjectNav-v1
2020-03-05 03:36:44,733 initializing sim Sim-v0
2020-03-05 03:36:48,449 Initializing task ObjectNav-v1
2020-03-05 03:36:49,552 distance_to_goal: 5.235198982556661
2020-03-05 03:36:49,552 spl: 0.0
But we should fix this by checking for the dataset in the path we say it should be in.
It appears that the point task file pointnav_gibson_v1.zip is missing the content directory for the val environments. Or perhaps I am missing something.
Thanks for any assistance.
May i ask when does the submission phase go live
Hi,
I have submitted a model to the test-challenge track.
Note: I am seeing the same "Submitted" status for the minival and test standard submissions. I am a bit concerned because I submitted one of the prior docker submissions using the aws link. I want to make sure this works as intended.
I submitted a docker image to the val_mini phase and test_standard phase a few hours ago.
It ran well locally on val_mini, but the server doesn't have any result yet for both phases. What may cause the problem?
BTW, do the submissions wait in a pending queue for evaluation or be evaluated immediately?
Thanks
Hi,
Is there any way to cancel a submission to evalai? I am not sure if our submission got uploaded correctly and would like to cancel and resubmit to make sure.
Hi,
I was evaluating the shortest path for the ObjectNav task. The second and third episode on val_mini seem to fail with the shortest path.
Looking at the path, it seems like the path is correct. The agent is navigating towards an instance of the target category, maybe there's some issue with success criteria?
Several problems.
Install nvidia-docker. Note: only supports Linux; no Windows or MacOS.
Do we need nvidia-docker v1 or v2?
./test_locally_objectnav_rgbd.sh --docker-name my_submission
Both test_locally_*.sh files were not executable (needed chmod a+x).
(base) dbatra@ubuntu-bionic-1:~/habitat-challenge$ ./test_locally_objectnav_rgbd.sh --docker-name my_submission
docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/create: dial unix /var/run/docker.sock: connect: permission denied.
See 'docker run --help'.
(base) dbatra@ubuntu-bionic-1:~/habitat-challenge$ sudo ./test_locally_objectnav_rgbd.sh --docker-name my_submission
docker: Error response from daemon: Unknown runtime specified nvidia.
See 'docker run --help'.
Could you kill my ongoing submission for test-challenge phase please so i can submit an update policy?
User karkus, Team: DAN, track: pointgoal; phase=test-ch.
Thank you!
I am getting the following error on both my minival and test-std submissions:
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 15.75 GiB total capacity; 146.35 MiB already allocated; 22.00 MiB free; 11.65 MiB cached)
Just to give context, my local runs take up < 4GB memory for evaluation. The above error also seems to suggest that only 146.35 + 22 + 11.65
MiB memory was available to begin with. Is it possible that multiple users are assigned to the same GPUs?
Hi
Based on the paper, I understand that the turn_angle for the agent has been changed to 30 degrees in 2020?
Why has this not been updated in the pointnav config file or the default config? Is there another configuration that I am missing?
Hi,
I found the API of habitat-challenge is different from the current version habitat-lab, can you specify which branch or version of habitat-lab shall I install to use habitat-challenge?
Thanks,
I followed the instructions to build the local docker file.
docker build . --file Pointnav_DDPPO_baseline.Dockerfile -t pointnav_submission_debug
It built successfully, but local testing via ./test_locally_pointnav_rgbd.sh
resulted in the following error:
Neither `ifconfig` (`ifconfig -a`) nor `ip` (`ip address show`) commands are available, listing network interfaces is likely to fail
2020-05-05 07:03:09,730 Overwriting CNN input size of depth: (256, 256)
2020-05-05 07:03:09,731 Overwriting CNN input size of rgb: (256, 256)
2020-05-05 07:03:12,762 Model checkpoint wasn't loaded, evaluating a random model.
2020-05-05 07:03:12,777 Initializing dataset PointNav-v1
2020-05-05 07:03:12,779 initializing sim Sim-v0
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0505 07:03:12.791268 16 WindowlessContext.cpp:114] Check failed: eglDevId < numDevices [EGL] Could not find an EGL device for CUDA device 0
*** Check failure stack trace: ***
submission.sh: line 3: 16 Aborted (core dumped) python agent.py --evaluation $AGENT_EVALUATION_TYPE $@
I created an interactive session inside the docker via:
docker run -v /tmp/habitat-challenge-data:/habitat-challenge-data --runtime=nvidia -it pointnav_submission_debug /bin/bash
nvidia-smi
worked:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.116.00 Driver Version: 418.116.00 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro GP100 Off | 00000000:81:00.0 Off | 0 |
| 26% 37C P0 31W / 235W | 0MiB / 16278MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Quadro GP100 Off | 00000000:82:00.0 Off | 0 |
| 26% 37C P0 29W / 235W | 0MiB / 16278MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
Running a simple pytorch code on the GPU also worked:
>>> python -c "import torch, torch.nn as nn; device = torch.device('cuda:0'); model = nn.Linear(4, 2); model.to(device); x = torch.randn(1, 4).to(device); y = model(x); print(y)"
tensor([[0.0405, 0.1198]], device='cuda:0', grad_fn=<AddmmBackward>)
I submitted my code successfully to the minival challenge phase:
evalai push [my docker image] --phase habitat21-objectnav-minival --private
My submission completed successfully, and I could view my results.
However, when I submit to the test-std challenge phase with the below command, I get the error copied at the bottom of my post, and the status of my submission is "Failed".
evalai push [my docker image] --phase habitat21-objectnav-test-std --private
I saw this same error on the minival phase the first time I submitted. I thought that the error meant that my code ran for too long, and the job was killed because I timed out. I sped up my code, resubmitted, and the error went away. Then, my submission successfully completed on the minival phase.
This error happens after my code runs for 30 minutes on the test-std phase. I previously saw this error after 30 minutes running on the minival phase. My understanding is that our submissions on the test-std phase have 48 hours to complete. There are no submissions visible yet on the test-std public leaderboard, so I do not know if anyone else ran their code successfully on this phase.
In summary, my questions are:
Thank you for any help you can provide!
Error message:
Traceback (most recent call last):
File "challenge_agent.py", line 308, in
main()
File "challenge_agent.py", line 304, in main
challenge.submit(agent)
File "/habitat-lab/habitat/core/challenge.py", line 19, in submit
metrics = super().evaluate(agent)
File "/habitat-lab/habitat/core/benchmark.py", line 163, in evaluate
return self.remote_evaluate(agent, num_episodes)
File "/habitat-lab/habitat/core/benchmark.py", line 93, in remote_evaluate
SerializedEntity=pack_for_grpc(action)
File "/opt/conda/envs/habitat/lib/python3.6/site-packages/grpcio-1.36.0rc1-py3.6-linux-x86_64.egg/grpc/_channel.py", line 923, in call
return _end_unary_response_blocking(state, call, False, None)
File "/opt/conda/envs/habitat/lib/python3.6/site-packages/grpcio-1.36.0rc1-py3.6-linux-x86_64.egg/grpc/_channel.py", line 826, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string = "{"created":"@1620152076.442809572","description":"Error received from peer ipv4:127.0.0.1:8085","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Socket closed","grpc_status":14}"
I have just tried to push my submission, however, soon (411.10 sec) I’ve got "Submission Status: failed"
. It seems that log files (stderr and stdout, as well as, results) are empty. Is there anything I can do to get more information regarding to this failure to debug my submission?
And yes, I tested my container with ./test_locally_{}.sh
To: habitat19-rgbd-minival
(duplicate from eval forum)
The configuration files provided in configs/challenge_pointnav2021.local.rgbd.yaml has
TURN_ANGLE: 30
but previous baselines like the DD-PPO paper use 10 degrees. I assume that for this challenge we have to use 30 right? The evaluation is performed with the same parameters as in configs/challenge_pointnav2021.local.rgbd.yaml ?
thanks
Hi!
I trained PPO agent model from habitat_baselines folder on pointnav task with gibson dataset. I wanted to evaluate trained model locally using docker. I created the following ppo agent submission script:
import argparse
import habitat
import random
import numpy
import os
from habitat.config import Config
from habitat.config.default import get_config
from habitat_baselines.agents.ppo_agents import PPOAgent
def get_default_config():
c = Config()
c.INPUT_TYPE = "rgbd"
c.MODEL_PATH = "models/ckpt.199.pth" # my trained model
c.RESOLUTION = 256
c.HIDDEN_SIZE = 512
c.RANDOM_SEED = 7
c.PTH_GPU_ID = 1
c.GOAL_SENSOR_UUID = "pointgoal"
return c
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--evaluation", type=str, required=True, choices=["local", "remote"])
args = parser.parse_args()
agent_config = get_default_config()
agent = PPOAgent(agent_config)
if args.evaluation == "local":
challenge = habitat.Challenge(eval_remote=False)
else:
challenge = habitat.Challenge(eval_remote=True)
challenge.submit(agent)
if __name__ == "__main__":
main()
I built the docker and after the command "sudo ./test_locally_pointnav_rgbd.sh --docker-name ppo_submission" got the following error:
2020-03-11 18:50:08,104 Initializing dataset PointNav-v1
2020-03-11 18:50:08,108 initializing sim Sim-v0
2020-03-11 18:50:09,020 Initializing task Nav-v0
Traceback (most recent call last):
File "ppo_agent.py", line 41, in <module>
main()
File "ppo_agent.py", line 37, in main
challenge.submit(agent)
File "/habitat-api/habitat/core/challenge.py", line 19, in submit
metrics = super().evaluate(agent)
File "/habitat-api/habitat/core/benchmark.py", line 159, in evaluate
return self.local_evaluate(agent, num_episodes)
File "/habitat-api/habitat/core/benchmark.py", line 133, in local_evaluate
action = agent.act(observations)
File "/habitat-api/habitat_baselines/agents/ppo_agents.py", line 134, in act
deterministic=False,
File "/habitat-api/habitat_baselines/rl/ppo/policy.py", line 40, in act
observations, rnn_hidden_states, prev_actions, masks
File "/opt/conda/envs/habitat/lib/python3.6/site-packages/torch-1.4.0-py3.6-linux-x86_64.egg/torch/nn/modules/module.py", line 532, in __call__
result = self.forward(*input, **kwargs)
File "/habitat-api/habitat_baselines/rl/ppo/policy.py", line 167, in forward
perception_embed = self.visual_encoder(observations)
File "/opt/conda/envs/habitat/lib/python3.6/site-packages/torch-1.4.0-py3.6-linux-x86_64.egg/torch/nn/modules/module.py", line 532, in __call__
result = self.forward(*input, **kwargs)
File "/habitat-api/habitat_baselines/rl/models/simple_cnn.py", line 147, in forward
return self.cnn(cnn_input)
File "/opt/conda/envs/habitat/lib/python3.6/site-packages/torch-1.4.0-py3.6-linux-x86_64.egg/torch/nn/modules/module.py", line 532, in __call__
result = self.forward(*input, **kwargs)
File "/opt/conda/envs/habitat/lib/python3.6/site-packages/torch-1.4.0-py3.6-linux-x86_64.egg/torch/nn/modules/container.py", line 100, in forward
input = module(input)
File "/opt/conda/envs/habitat/lib/python3.6/site-packages/torch-1.4.0-py3.6-linux-x86_64.egg/torch/nn/modules/module.py", line 532, in __call__
result = self.forward(*input, **kwargs)
File "/opt/conda/envs/habitat/lib/python3.6/site-packages/torch-1.4.0-py3.6-linux-x86_64.egg/torch/nn/modules/linear.py", line 87, in forward
return F.linear(input, self.weight, self.bias)
File "/opt/conda/envs/habitat/lib/python3.6/site-packages/torch-1.4.0-py3.6-linux-x86_64.egg/torch/nn/functional.py", line 1370, in linear
ret = torch.addmm(bias, input, weight.t())
RuntimeError: size mismatch, m1: [1 x 99712], m2: [25088 x 512] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:290
I am wondering what could cause such error? Please help)
Hi,
As the description said there is a subset of the original MP3D category list that only contains 21 items. I was wondering if it is publicly accessible? If it is, where we can find it?
Thanks!
I know this challenge is over, but I want to have a try.
I try the script for local evaluation with rgb.pth, I follow the instruction without changes. I find the result I run is 0.4626, but the result from the leaderboard is 0.47. Is that the same model between the repo and the leaderboard?
@mathfac @dhruvbatra Hi!
Another issue on my side I am struggling with is the right training of habitat baseline ppo model with 2020 challenge configuration for pointnav task using habitat-api.
As PPO agent configuration I use the following file ppo_pointnav.yaml
BASE_TASK_CONFIG_PATH: "configs/tasks/pointnav_gib_rgbd_2020.yaml"
TRAINER_NAME: "ppo"
ENV_NAME: "NavRLEnv"
SIMULATOR_GPU_ID: 1
TORCH_GPU_ID: 1
VIDEO_OPTION: ["disk", "tensorboard"]
TENSORBOARD_DIR: "tb"
VIDEO_DIR: "video_dir"
TEST_EPISODE_COUNT: 994
EVAL_CKPT_PATH_DIR: "data/ppo_2020_checkpoints"
NUM_PROCESSES: 4
SENSORS: ["DEPTH_SENSOR"]
CHECKPOINT_FOLDER: "data/ppo_2020_checkpoints"
NUM_UPDATES: 270000
LOG_INTERVAL: 25
CHECKPOINT_INTERVAL: 2000
RL:
PPO:
clip_param: 0.1
ppo_epoch: 4
num_mini_batch: 2
value_loss_coef: 0.5
entropy_coef: 0.01
lr: 2.5e-4
eps: 1e-5
max_grad_norm: 0.5
num_steps: 128
hidden_size: 512
use_gae: True
gamma: 0.99
tau: 0.95
use_linear_clip_decay: True
use_linear_lr_decay: True
reward_window_size: 50
And for task configuration I used the same parameters as in challenge_pointnav2020.local.rgbd.yaml file:
ENVIRONMENT:
MAX_EPISODE_STEPS: 500
SIMULATOR:
AGENT_0:
SENSORS: ['RGB_SENSOR', 'DEPTH_SENSOR']
HEIGHT: 0.88
RADIUS: 0.18
HABITAT_SIM_V0:
GPU_DEVICE_ID: 0
ALLOW_SLIDING: False
RGB_SENSOR:
WIDTH: 640
HEIGHT: 360
HFOV: 70
POSITION: [0, 0.88, 0]
NOISE_MODEL: "GaussianNoiseModel"
NOISE_MODEL_KWARGS:
intensity_constant: 0.1
DEPTH_SENSOR:
WIDTH: 640
HEIGHT: 360
HFOV: 70
MIN_DEPTH: 0.1
MAX_DEPTH: 10.0
POSITION: [0, 0.88, 0]
NOISE_MODEL: "RedwoodDepthNoiseModel"
ACTION_SPACE_CONFIG: 'pyrobotnoisy'
NOISE_MODEL:
ROBOT: "LoCoBot"
CONTROLLER: 'Proportional'
NOISE_MULTIPLIER: 0.5
TASK:
TYPE: Nav-v0
SUCCESS_DISTANCE: 0.36
SENSORS: ['POINTGOAL_SENSOR']
POINTGOAL_SENSOR:
GOAL_FORMAT: POLAR
DIMENSIONALITY: 2
GOAL_SENSOR_UUID: pointgoal
MEASUREMENTS: ['DISTANCE_TO_GOAL', "SUCCESS", 'SPL']
SUCCESS:
SUCCESS_DISTANCE: 0.36
Just changed path to the train dataset (Habitat Challenge Data for Gibson (1.5 GB)):
DATASET:
TYPE: PointNav-v1
SPLIT: train
DATA_PATH: data/datasets/pointnav/gibson/v1/{split}/{split}.json.gz
After runnig command python -u habitat_baselines/run.py --exp-config habitat_baselines/config/pointnav/ppo_pointnav.yaml --run-type
I got the following error:
---
The active scene does not contain semantic annotations.
---
I0325 20:17:08.559100 8915 simulator.py:143] Loaded navmesh data/scene_datasets/gibson/Monson.navmesh
I0325 20:17:08.559392 8915 simulator.py:155] Recomputing navmesh for agent's height 0.88 and radius 0.18.
I0325 20:17:08.567361 8915 PathFinder.cpp:338] Building navmesh with 275x112 cells
I0325 20:17:08.655342 8915 PathFinder.cpp:606] Created navmesh with 137 vertices 61 polygons
I0325 20:17:08.655371 8915 Simulator.cpp:403] reconstruct navmesh successful
2020-03-25 20:17:08,720 Initializing task Nav-v0
2020-03-25 20:17:11,725 agent number of parameters: 52694149
/home/pryhoda/anaconda3/envs/habitat/lib/python3.6/site-packages/torch-1.4.0-py3.6-linux-x86_64.egg/torch/optim/lr_scheduler.py:122: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`. Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [3,0,0], thread: [0,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [3,0,0], thread: [1,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [3,0,0], thread: [2,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [3,0,0], thread: [3,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [0,0,0], thread: [0,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [0,0,0], thread: [1,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [0,0,0], thread: [2,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [0,0,0], thread: [3,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [2,0,0], thread: [0,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [2,0,0], thread: [1,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [2,0,0], thread: [2,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [2,0,0], thread: [3,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [1,0,0], thread: [0,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [1,0,0], thread: [1,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [1,0,0], thread: [2,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [1,0,0], thread: [3,0,0] Assertion `val >= zero` failed.
Traceback (most recent call last):
File "habitat_baselines/run.py", line 70, in <module>
main()
File "habitat_baselines/run.py", line 40, in main
run_exp(**vars(args))
File "habitat_baselines/run.py", line 64, in run_exp
trainer.train()
File "/home/pryhoda/HabitatProject/habitat-api/habitat_baselines/rl/ppo/ppo_trainer.py", line 346, in train
rollouts, current_episode_reward, running_episode_stats
File "/home/pryhoda/HabitatProject/habitat-api/habitat_baselines/rl/ppo/ppo_trainer.py", line 181, in _collect_rollout_step
outputs = self.envs.step([a[0].item() for a in actions])
File "/home/pryhoda/HabitatProject/habitat-api/habitat_baselines/rl/ppo/ppo_trainer.py", line 181, in <listcomp>
outputs = self.envs.step([a[0].item() for a in actions])
RuntimeError: CUDA error: device-side assert triggered
Exception ignored in: <bound method VectorEnv.__del__ of <habitat.core.vector_env.VectorEnv object at 0x7f8dfa79ea58>>
Traceback (most recent call last):
File "/home/pryhoda/HabitatProject/habitat-api/habitat/core/vector_env.py", line 468, in __del__
self.close()
File "/home/pryhoda/HabitatProject/habitat-api/habitat/core/vector_env.py", line 350, in close
write_fn((CLOSE_COMMAND, None))
File "/home/pryhoda/anaconda3/envs/habitat/lib/python3.6/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/home/pryhoda/anaconda3/envs/habitat/lib/python3.6/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header + buf)
File "/home/pryhoda/anaconda3/envs/habitat/lib/python3.6/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
I am wondering if PPO agent is not adapted to train with 2020 challenge config (it run ok for me with 2019 challenge config - pointnav_gibson_rgbd.yaml ) or it is some issues on my side ? Thanks in advance!
Hi,
Is there an option to terminate a submission currently being executed?
If not, could you manually terminate all my 3 submissions in the test-std track under user name "karkus" or increase the max number of parallel submissions?
I believe the scripts are hanging because of an extra process i launch, and wont exit unless the container is terminated.
thanks!
Just wanted to make initial submission with DD-PPO file without own training.
After running command:
docker build . --file Pointnav_DDPPO_Baseline.Dockerfile -t pointnav_submission
get an error:
ADD failed: stat /var/lib/docker/tmp/docker-builder885297584/demo.ckpt.pth: no such file or directory
Where can I find demo.ckpt.pth
file for initial submission ?
Evalai submissions don't seem to be scheduled anymore. I have submissions to pointnav test-std and minival that have remained in "submitted" status for over a week now
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.