facebookresearch / habitat-challenge Goto Github PK

View Code? Open in Web Editor NEW

303.0 23.0 57.0 46.31 MB

Code for the habitat challenge

Home Page: https://aihabitat.org

License: MIT License

Dockerfile 16.79% Python 77.57% Shell 5.64%

habitat-challenge's Issues

Time limit clarifications

Hi,

Few clarification questions about runtime limits for the pointnav task

What is the time limit for test-standard and test-challenge phase? The README mentions 24 hours, but i have noticed in the logs "Timelimit for test-std, test-challenge track is 151200 seconds" (42 hours)
What is the number of evaluation episodes?
Unfortunately my solution is currently rather slow, taking around 2 minutes per episode on average. 2000 episodes would take around 66 hours. Is it possible to extend the time limit?
I have also noticed that if my script does not terminate within the time limit there is no output generated and the submission is marked failed. Would it be possible to generate results, maybe treating any remaining episodes as failed? I have done something like this in my script (always take stop action if total execution time is beyond limit), but this limits my runtime to a specified amount, and I cannot be sure how much extra time will be spent on loading remaining episodes

thank you

Missing cuda in docker for submission

My code requires this library for running: https://github.com/rusty1s/pytorch_scatter . When I try installing it, it throws out the following error:

  No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/sub.py -> build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/add.py -> build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/div.py -> build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/mul.py -> build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/std.py -> build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/__init__.py -> build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/mean.py -> build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/logsumexp.py -> build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/min.py -> build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/max.py -> build/lib.linux-x86_64-3.6/torch_scatter
  creating build/lib.linux-x86_64-3.6/test
  copying test/test_backward.py -> build/lib.linux-x86_64-3.6/test
  copying test/__init__.py -> build/lib.linux-x86_64-3.6/test
  copying test/test_logsumexp.py -> build/lib.linux-x86_64-3.6/test
  copying test/test_max_min.py -> build/lib.linux-x86_64-3.6/test
  copying test/utils.py -> build/lib.linux-x86_64-3.6/test
  copying test/test_std.py -> build/lib.linux-x86_64-3.6/test
  copying test/test_multi_gpu.py -> build/lib.linux-x86_64-3.6/test
  copying test/test_forward.py -> build/lib.linux-x86_64-3.6/test
  copying test/test_broadcasting.py -> build/lib.linux-x86_64-3.6/test
  creating build/lib.linux-x86_64-3.6/torch_scatter/composite
  copying torch_scatter/composite/__init__.py -> build/lib.linux-x86_64-3.6/torch_scatter/composite
  copying torch_scatter/composite/softmax.py -> build/lib.linux-x86_64-3.6/torch_scatter/composite
  creating build/lib.linux-x86_64-3.6/torch_scatter/utils
  copying torch_scatter/utils/ext.py -> build/lib.linux-x86_64-3.6/torch_scatter/utils
  copying torch_scatter/utils/__init__.py -> build/lib.linux-x86_64-3.6/torch_scatter/utils
  copying torch_scatter/utils/gen.py -> build/lib.linux-x86_64-3.6/torch_scatter/utils
  running build_ext
  building 'torch_scatter.scatter_cpu' extension
  creating build/temp.linux-x86_64-3.6
  creating build/temp.linux-x86_64-3.6/cpu
  gcc -pthread -B /opt/conda/envs/habitat/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/TH -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/THC -I/opt/conda/envs/habitat/include/python3.6m -c cpu/scatter.cpp -o build/temp.linux-x86_64-3.6/cpu/scatter.o -Wno-unused-variable -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=scatter_cpu -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
  cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
  g++ -pthread -shared -B /opt/conda/envs/habitat/compiler_compat -L/opt/conda/envs/habitat/lib -Wl,-rpath=/opt/conda/envs/habitat/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/cpu/scatter.o -o build/lib.linux-x86_64-3.6/torch_scatter/scatter_cpu.cpython-36m-x86_64-linux-gnu.so
  building 'torch_scatter.scatter_cuda' extension
  creating build/temp.linux-x86_64-3.6/cuda
  gcc -pthread -B /opt/conda/envs/habitat/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/TH -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/envs/habitat/include/python3.6m -c cuda/scatter.cpp -o build/temp.linux-x86_64-3.6/cuda/scatter.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=scatter_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
  cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
  /usr/local/cuda/bin/nvcc -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/TH -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/envs/habitat/include/python3.6m -c cuda/scatter_kernel.cu -o build/temp.linux-x86_64-3.6/cuda/scatter_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=scatter_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
  unable to execute '/usr/local/cuda/bin/nvcc': No such file or directory
  error: command '/usr/local/cuda/bin/nvcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for torch-scatter

Looks like nvcc is missing. I tried manually looking at the base docker. I couldn't find nvcc. Should I manually install it in that case?

Intermittent Failing/Finished Status of Identical EvalAI Submissions

I submitted to ObjectNav Test-Challenge. On EvalAI, the submission turned up as Cancelled almost immediately after submitting. The submission appeared again (about 10 min later) as a second submission on the site and was Running for 229350 sec. This job ultimately ended with the status Failed with blank result, std out and std err files.

I submitted exactly the same container to the minival phase 1 time and the Test-Std phase 2 times. The minival submission completed successfully with results. The first Test-Std submission ended with a Failed status and blank result, std out and std err files. The second Test-Std submission completed successfully with results.

Do you have any ideas why these submissions could be intermittently Failing?
Is there a way I can try to get my submission to the Test-Challenge phase running without using my 6 total submissions since I have successfully run this submission on both of the other phases?

Remote evaluations fail

Hi, I have made a couple of remote submissions recently (pointnav test-std) but they all fail. I have seen two types of failure cases, both happened multiple times.

1, script fails after a few hours, and it produces an stderr output with the following error:

Traceback (most recent call last):
  File "agent.py", line 172, in <module>
    main()
  File "agent.py", line 168, in main
    challenge.submit(agent)
  File "/habitat-api/habitat/core/challenge.py", line 19, in submit
    metrics = super().evaluate(agent)
  File "/habitat-api/habitat/core/benchmark.py", line 171, in evaluate
    return self.remote_evaluate(agent, num_episodes)
  File "/habitat-api/habitat/core/benchmark.py", line 103, in remote_evaluate
    SerializedEntity=pack_for_grpc(action)
  File "/opt/conda/envs/habitat/lib/python3.6/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/opt/conda/envs/habitat/lib/python3.6/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "Socket closed"
	debug_error_string = "{"created":"@1599858112.185563913","description":"Error received from peer ipv4:127.0.0.1:8085","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket closed","grpc_status":14}"

2, the script fails after 42 hours and it produces empty stdout and stderr files.

Unable to make new submissions on the EvalAI site

For the past two days I have been unable to make new submissions to the EvalAI site. I even re-cloned the repo, cleared my docker caches and tried to resubmit the random baseline agent with no success.

Previously I was able to make submissions, so I am wondering if the issue is on the EvalAI platform?

I created a topic on their forum but I have had no response yet, if you are able to confirm that the platform is still working that would be much appreciated. (for example by submitting a random agent)

Inconsistent scores between Local and Remote minival for PointNav

I thought it was better to raise a separate issue for this. I'm observing large inconsistencies in the local and remote submissions. I made 3 submissions of the same model on the minival track. I also evaluated the docker on a local server. These were the results I got:

Local docker evaluation

	SPL	Success	Distance to goal
Trial 1	0.170	0.244	2.630
Trial 2	0.150	0.206	2.491
Trial 3	0.172	0.234	2.486

Remote docker evaluation

	SPL	Success	Distance to goal
Trial 1	0.260	0.366	0.895
Trial 2	0.258	0.333	0.846
Trial 3	0.336	0.433	0.870

I additionally evaluated locally on a ppo_trainer evaluation script I wrote based in habitat-baselines. I've tried to keep things as consistent as possible between my docker submission and the ppo_trainer script.

Local non-docker evaluation

	SPL	Success	Distance to goal
Trial 1	0.327	0.455	1.504
Trial 2	0.350	0.492	1.503
Trial 3	0.333	0.455	1.622

Could these inconsistencies be due to some random seed issues? I've set them to 123 as follows:

random.seed(config.PYT_RANDOM_SEED)
np.random.seed(config.PYT_RANDOM_SEED)
torch.random.manual_seed(config.PYT_RANDOM_SEED)
if torch.cuda.is_available():
   torch.backends.cudnn.deterministic = True
   torch.backends.cudnn.benchmark = False

Clarification needed regarding the submission deadline

Is it sufficient for our final models to be under submission before the deadline, or should they have strictly finished evaluating before the deadline?

Inconsistent Scores Between Local and Remote minival for ObjectNav

I followed the submission criteria and made a docker submission for minival. Locally, it consistently reaches a given distance to goal, but remotely, it receives a different, higher distance to goal.

(where the 6.63 is distance_to_goal)
Is this noise? I'm surprised, as I received the same results on different machines, and also different runs of the same container (as shown in first screenshot).

Cannot run tests, wrong checkpoint? torch.Size([128, 1024, 3, 3]) from checkpoint, the shape in current model is torch.Size([82, 1024, 3, 3]).

I followed these instructions on ubuntu:
https://colab.research.google.com/gist/mathfac/8c9b97d7afef36e377f17d587c903ede#scrollTo=Qo_z277BAueV

But when I run the test I get:

(base) ggg:~/habitat/habitat-challenge$ bash ./test_locally_pointnav_rgbd.sh --docker-name ddppo_pointnav_submission
Neither `ifconfig` (`ifconfig -a`) nor `ip` (`ip address show`) commands are available, listing network interfaces is likely to fail
Checkpoint loaded: demo.ckpt.pth
/habitat-lab/habitat_baselines/config/default.py:214: UserWarning: NUM_PROCESSES is depricated and will be removed in a future version.  Use NUM_ENVIRONMENTS instead.  Overwriting NUM_ENVIRONMENTS with NUM_PROCESSES for backwards compatibility.
  "NUM_PROCESSES is depricated and will be removed in a future version."
Traceback (most recent call last):
  File "agent.py", line 200, in <module>
    main()
  File "agent.py", line 189, in main
    agent = DDPPOAgent(config)
  File "agent.py", line 124, in __init__
    for k, v in ckpt["state_dict"].items()
  File "/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/nn/modules/module.py", line 847, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for PointNavResNetPolicy:
	Missing key(s) in state_dict: "net.pointgoal_embedding.weight", "net.pointgoal_embedding.bias". 
	Unexpected key(s) in state_dict: "net.tgt_embeding.weight", "net.tgt_embeding.bias". 
	size mismatch for net.visual_encoder.compression.0.weight: copying a param with shape torch.Size([128, 1024, 3, 3]) from checkpoint, the shape in current model is torch.Size([82, 1024, 3, 3]).
	size mismatch for net.visual_encoder.compression.1.weight: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([82]).
	size mismatch for net.visual_encoder.compression.1.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([82]).
	size mismatch for net.visual_fc.1.weight: copying a param with shape torch.Size([512, 2048]) from checkpoint, the shape in current model is torch.Size([512, 2050]).

Any suggestion?

Submission failed on test standard

Hi,
Our submission keeps failing on test standard with "standard_error", which I am assuming is because of time limit. The same docker images takes 14mins for remote evaluation on val_mini. Assuming test standard is 33-66 times val_mini (1000-2000 episodes vs 30 episodes), our submission should take 8-16 hrs on test standard. But the submission ran for 42 hrs and failed. Is there any way to get more information such as how many episodes were completed and left, average length of episodes, the intermediate scores when the submission timed out, etc?

Any other suggestions on how to figure out the problem would be helpful.

Thanks

Remote evaluation on "submitted" from 5 days

Hi, I've made a submission for the ObjectNav task in the test-std phase. Currently is in "submitted" state for about 5 days. Is this due to a switch from the 2020 Challenge to the 2021 Challenge? Or are there some problems with the remote server?

Thanks,
Tommaso

Are the 2021 competition fairembodied/habitat-challenge docker images all published?

Troubleshooting why I could not run and try your 2021 docker images for the habitat-challenge today and it seems like they might not have been published yet on docker hub? Or am I missing something? I'm able to docker pull fairembodied/habitat-challenge:latest but trying to build from the dockerfiles and that tag causes errors.

docker pull fairembodied/habitat-challenge:testing_2021_habitat_base_docker

Error response from daemon: manifest for fairembodied/habitat-challenge:testing_2021_habitat_base_docker not found: manifest unknown: manifest unknown

thanks.

Unable to run PointNav agent locally

While running the "test_locally_pointnav_rgbd.sh" script, I get the following error:

2020-03-24 10:15:01,170 Initializing dataset PointNav-v1
2020-03-24 10:15:01,171 initializing sim Sim-v0
2020-03-24 10:15:02,310 Initializing task Nav-v0
Traceback (most recent call last):
  File "agent.py", line 26, in <module>
    main()
  File "agent.py", line 21, in main
    challenge = habitat.Challenge()
  File "/habitat-api/habitat/core/challenge.py", line 16, in __init__
    super().__init__(config_paths)
  File "/habitat-api/habitat/core/benchmark.py", line 30, in __init__
    self._env = Env(config=config_env)
  File "/habitat-api/habitat/core/env.py", line 102, in __init__
    dataset=self._dataset,
  File "/habitat-api/habitat/tasks/registration.py", line 21, in make_task
    return _task(**kwargs)
  File "/habitat-api/habitat/tasks/nav/nav.py", line 968, in __init__
    super().__init__(config=config, sim=sim, dataset=dataset)
  File "/habitat-api/habitat/core/embodied_task.py", line 241, in __init__
    entities_config=config,
  File "/habitat-api/habitat/core/embodied_task.py", line 269, in _init_entities
    entity_type = register_func(entity_cfg.TYPE)
  File "/opt/conda/envs/habitat/lib/python3.6/site-packages/yacs/config.py", line 141, in __getattr__
    raise AttributeError(name)
AttributeError: TYPE

When I traced the error, I found that the task type specified in "challenge_pointnav2020.local.rgbd.yaml" file is not being read by yacs. Please let me know how I can fix this problem.

EvalAI Submission Instructions

In the submission instructions it states that valid phases are: habitat21-{pointnav, objectnav}-{minival, test-std, test-ch}. However, on the eval.ai submission website, the submission phases are listed in the format: habitat-objectnav-minival-2021-802.

When trying to submit with the github instructions, I get the error:

Error: Challenge phase with slug habitat21-objectnav-minival does not exist

When trying to submit with the phases listed in the eval.ai website, I get the error:

Error: Sorry, cannot accept submissions since challenge phase is not active

Note that I can successfully make 2020 minival submissions on eval.ai.

Are the correct phases activated? Also, what is the correct phase naming convention?

Thank you!

Issue with running a random agent.

"standard_error" on submission

Hi,
My submission to the evalai portal failed and it is showing just "standard_error" in the Stderr File.
Could you help in figuring out the error?

How can I evaluate without the docker?

Seems this code needs to build docker, how can we run it under the official habitat-api / habitat-sim environment without the docker?

Thanks a lot!

Docker installation with data softlink bug fixed?

Hi there,

I found that in current ReadMe file local evaluation step 5, if a soft link is used and the suggested docker command is run,
there would be an error message like this:

 2021-04-13 21:51:15,680 Initializing dataset ObjectNav-v1
 2021-04-13 21:51:15,710 initializing sim Sim-v0
 WARNING: Logging before InitGoogleLogging() is written to STDERR
 E0413 21:51:15.723701    17 StageAttributesManager.cpp:90] StageAttributesManager::registerObjectFinalize : Render asset template handle : habitat-challenge-data/data/scene_datasets/mp3d/x8F5xyUWy9e/x8F5xyUWy9e.glb specified in stage template with handle : habitat-challenge-data/data/scene_datasets/mp3d/x8F5xyUWy9e/x8F5xyUWy9e.glb does not correspond to any existing file or primitive render asset.  Aborting. 
  submission.sh: line 3:    17 Segmentation fault      (core dumped) python agent.py --evaluation $AGENT_EVALUATION_TYPE $@`

My fix was to change this line:

 -v $(realpath habitat-challenge-data/data/scene_datasets/mp3d) \

  -v $(realpath habitat-challenge-data/data/scene_datasets/mp3d):/habitat-challenge-data/data/scene_datasets/mp3d \

Not that familiar with Docker, so not sure whether this is a version problem or a bug.
Just want to post one solution!

Best,
Yiqing

is that possible to submit for test standard or minival now

hi, I try to submit my result for minival, the status shows "submitted", but execution time and the result are None. I guess there may be some mistakes in my docker. So I follow Readme to submit the original agent and ppo agent. The status is "submitted" and others show "None".

My question is that is it possible to submit for test or minival. If so, any suggestions for my problems.

Thanks a lot.

SPL agent path length calculation

How is the agent path length calculated in the SPL metric?

No such file or directory: 'habitat-challenge-data/objectgoal_mp3d/val_mini/val_mini.json.gz'

Hi. I am trying to run the test_locally_objectnav_rgbd.sh script with the provided RandomAgent. I am following the Local Evaluation instruction:

Cloned the master branch
Built the objectnav_submission image using

docker build . --file Objectnav.Dockerfile -t objectnav_submission

Nvidia-docker installed and working
No modifications to the Dockerfile. Using all defaults.
This is my directory structure:

tree habitat-challenge-data
habitat-challenge-data
├── data
│   ├── datasets
│   │   └── objectnav
│   │       └── mp3d
│   │           └── v1
│   │               ├── objectnav_mp3d_v1.zip
│   │               ├── train
│   │               │   ├── content
│   │               │   │   ├── 17DRP5sb8fy.json.gz
│   │               │   │   ├── 1LXtFkjw3qL.json.gz
│   │               │   │   ├── 1pXnuDYAj8r.json.gz
│   │               │   │   ├── 29hnd4uzFmX.json.gz
│   │               │   │   ├── 5LpN3gDmAk7.json.gz
│   │               │   │   ├── 5q7pvUzZiYa.json.gz
│   │               │   │   ├── 759xd9YjKW5.json.gz
│   │               │   │   ├── 7y3sRwLe3Va.json.gz
│   │               │   │   ├── 82sE5b5pLXE.json.gz
│   │               │   │   ├── 8WUmhLawc2A.json.gz
│   │               │   │   ├── aayBHfsNo7d.json.gz
│   │               │   │   ├── ac26ZMwG7aT.json.gz
│   │               │   │   ├── B6ByNegPMKs.json.gz
│   │               │   │   ├── b8cTxDM8gDG.json.gz
│   │               │   │   ├── cV4RVeZvu5T.json.gz
│   │               │   │   ├── D7G3Y4RVNrH.json.gz
│   │               │   │   ├── D7N2EKCX4Sj.json.gz
│   │               │   │   ├── dhjEzFoUFzH.json.gz
│   │               │   │   ├── E9uDoFAP3SH.json.gz
│   │               │   │   ├── e9zR4mvMWw7.json.gz
│   │               │   │   ├── EDJbREhghzL.json.gz
│   │               │   │   ├── GdvgFV5R1Z5.json.gz
│   │               │   │   ├── gZ6f7yhEvPG.json.gz
│   │               │   │   ├── HxpKQynjfin.json.gz
│   │               │   │   ├── i5noydFURQK.json.gz
│   │               │   │   ├── JeFG25nYj2p.json.gz
│   │               │   │   ├── JF19kD82Mey.json.gz
│   │               │   │   ├── jh4fc5c5qoQ.json.gz
│   │               │   │   ├── kEZ7cmS4wCh.json.gz
│   │               │   │   ├── mJXqzFtmKg4.json.gz
│   │               │   │   ├── p5wJjkQkbXX.json.gz
│   │               │   │   ├── Pm6F8kyY3z2.json.gz
│   │               │   │   ├── pRbA3pwrgk9.json.gz
│   │               │   │   ├── PuKPg4mmafe.json.gz
│   │               │   │   ├── PX4nDJXEHrG.json.gz
│   │               │   │   ├── qoiz87JEwZ2.json.gz
│   │               │   │   ├── r1Q1Z4BcV1o.json.gz
│   │               │   │   ├── r47D5H71a5s.json.gz
│   │               │   │   ├── rPc6DW4iMge.json.gz
│   │               │   │   ├── s8pcmisQ38h.json.gz
│   │               │   │   ├── S9hNv5qa7GM.json.gz
│   │               │   │   ├── sKLMLpTHeUy.json.gz
│   │               │   │   ├── sT4fr6TAbpF.json.gz
│   │               │   │   ├── ULsKaCPVFJR.json.gz
│   │               │   │   ├── uNb9QFRL6hY.json.gz
│   │               │   │   ├── ur6pFq6Qu1A.json.gz
│   │               │   │   ├── Uxmj2M2itWa.json.gz
│   │               │   │   ├── V2XKFyX4ASd.json.gz
│   │               │   │   ├── VFuaQ6m2Qom.json.gz
│   │               │   │   ├── VLzqgDo317F.json.gz
│   │               │   │   ├── VVfe2KiqLaN.json.gz
│   │               │   │   ├── Vvot9Ly1tCj.json.gz
│   │               │   │   ├── vyrNrziPKCB.json.gz
│   │               │   │   ├── XcA2TqTSSAj.json.gz
│   │               │   │   ├── YmJkqBEsHnH.json.gz
│   │               │   │   └── ZMojNkEp431.json.gz
│   │               │   └── train.json.gz
│   │               ├── val
│   │               │   ├── content
│   │               │   │   ├── 2azQ1b91cZZ.json.gz
│   │               │   │   ├── 8194nk5LbLH.json.gz
│   │               │   │   ├── EU6Fwq7SyZv.json.gz
│   │               │   │   ├── oLBMNvg9in8.json.gz
│   │               │   │   ├── pLe4wQe7qrG.json.gz
│   │               │   │   ├── QUCTc6BB5sX.json.gz
│   │               │   │   ├── TbHJrupSAjP.json.gz
│   │               │   │   ├── X7HyMhZNoso.json.gz
│   │               │   │   ├── x8F5xyUWy9e.json.gz
│   │               │   │   ├── Z6MFQCViBuw.json.gz
│   │               │   │   └── zsNo4HB9uLZ.json.gz
│   │               │   └── val.json.gz
│   │               └── val_mini
│   │                   └── val_mini.json.gz
│   └── scene_datasets
│       └── mp3d
│           ├── download_mp.py
│           └── v1
│               └── scans
│                   └── 17DRP5sb8fy
│                       ├── cameras.zip
│                       ├── matterport_camera_intrinsics.zip
│                       ├── matterport_camera_poses.zip
│                       ├── matterport_color_images.zip
│                       └── tmp0PJfol
├── objectgoal_mp3d
│   └── val_mini
│       └── val_mini.json.gz
└── pointgoal_gibson_v2
    └── val_mini
        ├── content
        │   └── Pablo.json.gz
        └── val_mini.json.gz

Running the evaluation script:

❯ ./test_locally_objectnav_rgbd.sh --docker-name objectnav_submission

2021-03-19 13:32:25,265 Initializing dataset ObjectNav-v1
Traceback (most recent call last):
  File "agent.py", line 41, in <module>
    main()
  File "agent.py", line 33, in main
    challenge = habitat.Challenge(eval_remote=False)
  File "/habitat-lab/habitat/core/challenge.py", line 16, in __init__
    super().__init__(config_paths, eval_remote=eval_remote)
  File "/habitat-lab/habitat/core/benchmark.py", line 38, in __init__
    self._env = Env(config=config_env)
  File "/habitat-lab/habitat/core/env.py", line 79, in __init__
    id_dataset=config.DATASET.TYPE, config=config.DATASET
  File "/habitat-lab/habitat/datasets/registration.py", line 20, in make_dataset
    return _dataset(**kwargs)  # type: ignore
  File "/habitat-lab/habitat/datasets/object_nav/object_nav_dataset.py", line 74, in __init__
    super().__init__(config)
  File "/habitat-lab/habitat/datasets/pointnav/pointnav_dataset.py", line 93, in __init__
    with gzip.open(datasetfile_path, "rt") as f:
  File "/opt/conda/envs/habitat/lib/python3.6/gzip.py", line 53, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
  File "/opt/conda/envs/habitat/lib/python3.6/gzip.py", line 163, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'habitat-challenge-data/objectnav_mp3d/val_mini/val_mini.json.gz'

Thanks for your help with this!

Socket timeout issue while doing evalai push

I ran the following command for evalai push
evalai push gp_nav_rgbd:latest --phase habitat19-rgbd-minival

and got the following error:

Traceback (most recent call last):
  File "/home/dchaplot/anaconda3/lib/python3.6/site-packages/urllib3/response.py", line 302, in _error_catcher
    yield
  File "/home/dchaplot/anaconda3/lib/python3.6/site-packages/urllib3/response.py", line 384, in read
    data = self._fp.read(amt)
  File "/home/dchaplot/anaconda3/lib/python3.6/http/client.py", line 449, in read
    n = self.readinto(b)
  File "/home/dchaplot/anaconda3/lib/python3.6/http/client.py", line 483, in readinto
    return self._readinto_chunked(b)
  File "/home/dchaplot/anaconda3/lib/python3.6/http/client.py", line 578, in _readinto_chunked
    chunk_left = self._get_chunk_left()
  File "/home/dchaplot/anaconda3/lib/python3.6/http/client.py", line 546, in _get_chunk_left
    chunk_left = self._read_next_chunk_size()
  File "/home/dchaplot/anaconda3/lib/python3.6/http/client.py", line 506, in _read_next_chunk_size
    line = self.fp.readline(_MAXLINE + 1)
  File "/home/dchaplot/anaconda3/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

My docker image is 7.67GB.

What might be the issue here?

New habitat-gibson dataset link

Hi,

it looks like the Gibson dataset link in Readme is broken, it is caused by that form has accepted maximum number of responses. Can you update to this one instead: https://forms.gle/36TW9uVpjrE1Mkf9A

Thanks a lot!

Submission status

Hi @abhiskk !
I made submissions for point nav minival and test-standart phases about 24 hours ago (UCULab team) and the status for those submissions is still "Running". What could cause such a long evaluation ?

Challenge resolution

Hello, thank you for creating this years challenge!

Regarding the resolution, for object-nav the config file has a resolution of WIDTH: 640, HEIGHT: 480. Will it be possible to use lower resolutions for the challenge? I believe this was available last year but wish to confirm.

Thank you.

Configuration clarifications

I have a couple of questions regarding the configurations:

A new file has been added: configs/challenge_pointnav2020_v2.local.rgbd.yaml which has a different sensor orientation. What is this for? Should we re-train our agents with this new configuration?
What is the config file configs/challenge_pointnav2020.local.rgbd_test_scene.yaml for? It is using v1 data. Should it not be v2?

Evaluation time-limits

How strictly will the model evaluation time be taken into consideration? I can imagine some models taking several hours rather than 30 minutes to complete 1000+ episodes.

Path error

Our instructions say:

a) PoinNav: Download Gibson scenes used for Habitat Challenge. Accept terms here and select the download corresponding to “Habitat Challenge Data for Gibson (1.5 GB)“. Place this data in: habitat-challenge/habitat-challenge-data/gibson

b) ObjectNav: Download Matterport3D scenes used for Habitat Challenge here. Place this data in: habitat-challenge/habitat-challenge-data/mp3d

But our scripts are looking for habitat-challenge/habitat-challenge-data/scene_datasets/mp3d, as you can see from these errors:

(base) dbatra@ubuntu-bionic-1:~/habitat-challenge$ ./test_locally_objectnav_rgbd.sh --docker-name my_submission
2020-03-05 03:15:30,744 Initializing dataset ObjectNav-v1
2020-03-05 03:15:30,776 initializing sim Sim-v0
WARNING: Logging before InitGoogleLogging() is written to STDERR
E0305 03:15:31.245062    16 ResourceManager.cpp:84] Cannot load from file habitat-challenge-data/scene_datasets/mp3d/pLe4wQe7qrG/pLe4wQe7qrG.glb
E0305 03:15:31.249548    16 Simulator.cpp:106] cannot load habitat-challenge-data/scene_datasets/mp3d/pLe4wQe7qrG/pLe4wQe7qrG.glb

If I create appropriate symlinks, this error goes away:

(base) dbatra@ubuntu-bionic-1:~/habitat-challenge$ ./test_locally_objectnav_rgbd.sh --docker-name my_submission
2020-03-05 03:36:44,704 Initializing dataset ObjectNav-v1
2020-03-05 03:36:44,733 initializing sim Sim-v0
2020-03-05 03:36:48,449 Initializing task ObjectNav-v1
2020-03-05 03:36:49,552 distance_to_goal: 5.235198982556661
2020-03-05 03:36:49,552 spl: 0.0

But we should fix this by checking for the dataset in the path we say it should be in.

Missing content directory from validation file

It appears that the point task file pointnav_gibson_v1.zip is missing the content directory for the val environments. Or perhaps I am missing something.

Thanks for any assistance.

submission phase

May i ask when does the submission phase go live

Clarifications about the challenge submission

Hi,

I have submitted a model to the test-challenge track.

The status is "Submitted". What exactly does that mean?
How do we know that the test challenge evaluation succeeded, and that things went as expected?
Given that we have upto 5 submissions, how will the best results be selected? Is it automatically based on the maximum SPL score, or do we have to somehow choose the final model?

Note: I am seeing the same "Submitted" status for the minival and test standard submissions. I am a bit concerned because I submitted one of the prior docker submissions using the aws link. I want to make sure this works as intended.

About online evaluations

I submitted a docker image to the val_mini phase and test_standard phase a few hours ago.

It ran well locally on val_mini, but the server doesn't have any result yet for both phases. What may cause the problem?

BTW, do the submissions wait in a pending queue for evaluation or be evaluated immediately?

Thanks

Cancelling submission to evalai

Hi,
Is there any way to cancel a submission to evalai? I am not sure if our submission got uploaded correctly and would like to cancel and resubmit to make sure.

Potential issue with shortest path or success criteria in ObjectNav

Hi,
I was evaluating the shortest path for the ObjectNav task. The second and third episode on val_mini seem to fail with the shortest path.

Looking at the path, it seems like the path is correct. The agent is navigating towards an instance of the target category, maybe there's some issue with success criteria?

Testing 2020 starter code

Several problems.

Install nvidia-docker. Note: only supports Linux; no Windows or MacOS.

Do we need nvidia-docker v1 or v2?

./test_locally_objectnav_rgbd.sh --docker-name my_submission

Both test_locally_*.sh files were not executable (needed chmod a+x).

Docker commands failed -- I suspect because of incorrect naming in submissions.sh

(base) dbatra@ubuntu-bionic-1:~/habitat-challenge$ ./test_locally_objectnav_rgbd.sh --docker-name my_submission
docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/create: dial unix /var/run/docker.sock: connect: permission denied.
See 'docker run --help'.

(base) dbatra@ubuntu-bionic-1:~/habitat-challenge$ sudo ./test_locally_objectnav_rgbd.sh --docker-name my_submission
docker: Error response from daemon: Unknown runtime specified nvidia.
See 'docker run --help'.

Last minute cancellation request

Could you kill my ongoing submission for test-challenge phase please so i can submit an update policy?
User karkus, Team: DAN, track: pointgoal; phase=test-ch.
Thank you!

CUDA out of memory error

I am getting the following error on both my minival and test-std submissions:

RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 15.75 GiB total capacity; 146.35 MiB already allocated; 22.00 MiB free; 11.65 MiB cached)

Just to give context, my local runs take up < 4GB memory for evaluation. The above error also seems to suggest that only 146.35 + 22 + 11.65 MiB memory was available to begin with. Is it possible that multiple users are assigned to the same GPUs?

Is the turn angle supposed to be 30 degrees?

Based on the paper, I understand that the turn_angle for the agent has been changed to 30 degrees in 2020?
Why has this not been updated in the pointnav config file or the default config? Is there another configuration that I am missing?

API verison

Hi,

I found the API of habitat-challenge is different from the current version habitat-lab, can you specify which branch or version of habitat-lab shall I install to use habitat-challenge?

Thanks,

EGL errors in docker

I followed the instructions to build the local docker file.

docker build . --file Pointnav_DDPPO_baseline.Dockerfile -t pointnav_submission_debug

It built successfully, but local testing via ./test_locally_pointnav_rgbd.sh resulted in the following error:

Neither `ifconfig` (`ifconfig -a`) nor `ip` (`ip address show`) commands are available, listing network interfaces is likely to fail
2020-05-05 07:03:09,730 Overwriting CNN input size of depth: (256, 256)
2020-05-05 07:03:09,731 Overwriting CNN input size of rgb: (256, 256)
2020-05-05 07:03:12,762 Model checkpoint wasn't loaded, evaluating a random model.
2020-05-05 07:03:12,777 Initializing dataset PointNav-v1
2020-05-05 07:03:12,779 initializing sim Sim-v0
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0505 07:03:12.791268    16 WindowlessContext.cpp:114] Check failed: eglDevId < numDevices [EGL] Could not find an EGL device for CUDA device 0
*** Check failure stack trace: ***
submission.sh: line 3:    16 Aborted                 (core dumped) python agent.py --evaluation $AGENT_EVALUATION_TYPE $@

I created an interactive session inside the docker via:

docker run -v /tmp/habitat-challenge-data:/habitat-challenge-data --runtime=nvidia -it pointnav_submission_debug /bin/bash

nvidia-smi worked:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.116.00   Driver Version: 418.116.00   CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro GP100        Off  | 00000000:81:00.0 Off |                    0 |
| 26%   37C    P0    31W / 235W |      0MiB / 16278MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Quadro GP100        Off  | 00000000:82:00.0 Off |                    0 |
| 26%   37C    P0    29W / 235W |      0MiB / 16278MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Running a simple pytorch code on the GPU also worked:

>>> python -c "import torch, torch.nn as nn; device = torch.device('cuda:0'); model = nn.Linear(4, 2); model.to(device);  x = torch.randn(1, 4).to(device); y = model(x); print(y)"

tensor([[0.0405, 0.1198]], device='cuda:0', grad_fn=<AddmmBackward>)

test-std Phase Submission Error (code runs successfully on minival)

I submitted my code successfully to the minival challenge phase:

evalai push [my docker image] --phase habitat21-objectnav-minival --private

My submission completed successfully, and I could view my results.

However, when I submit to the test-std challenge phase with the below command, I get the error copied at the bottom of my post, and the status of my submission is "Failed".

evalai push [my docker image] --phase habitat21-objectnav-test-std --private

I saw this same error on the minival phase the first time I submitted. I thought that the error meant that my code ran for too long, and the job was killed because I timed out. I sped up my code, resubmitted, and the error went away. Then, my submission successfully completed on the minival phase.

This error happens after my code runs for 30 minutes on the test-std phase. I previously saw this error after 30 minutes running on the minival phase. My understanding is that our submissions on the test-std phase have 48 hours to complete. There are no submissions visible yet on the test-std public leaderboard, so I do not know if anyone else ran their code successfully on this phase.

In summary, my questions are:

Does this error indicate a timeout for my job?
If so, how long does our code have to run on the test-std phase?
If not, do you have any insight on what this error indicates?

Thank you for any help you can provide!

Error message:

Traceback (most recent call last):
File "challenge_agent.py", line 308, in
main()
File "challenge_agent.py", line 304, in main
challenge.submit(agent)
File "/habitat-lab/habitat/core/challenge.py", line 19, in submit
metrics = super().evaluate(agent)
File "/habitat-lab/habitat/core/benchmark.py", line 163, in evaluate
return self.remote_evaluate(agent, num_episodes)
File "/habitat-lab/habitat/core/benchmark.py", line 93, in remote_evaluate
SerializedEntity=pack_for_grpc(action)
File "/opt/conda/envs/habitat/lib/python3.6/site-packages/grpcio-1.36.0rc1-py3.6-linux-x86_64.egg/grpc/_channel.py", line 923, in call
return _end_unary_response_blocking(state, call, False, None)
File "/opt/conda/envs/habitat/lib/python3.6/site-packages/grpcio-1.36.0rc1-py3.6-linux-x86_64.egg/grpc/_channel.py", line 826, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string = "{"created":"@1620152076.442809572","description":"Error received from peer ipv4:127.0.0.1:8085","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Socket closed","grpc_status":14}"

Submission failed without any trace

I have just tried to push my submission, however, soon (411.10 sec) I’ve got "Submission Status: failed". It seems that log files (stderr and stdout, as well as, results) are empty. Is there anything I can do to get more information regarding to this failure to debug my submission?

And yes, I tested my container with ./test_locally_{}.sh
To: habitat19-rgbd-minival

(duplicate from eval forum)

turn angle 30 degrees

The configuration files provided in configs/challenge_pointnav2021.local.rgbd.yaml has
TURN_ANGLE: 30
but previous baselines like the DD-PPO paper use 10 degrees. I assume that for this challenge we have to use 30 right? The evaluation is performed with the same parameters as in configs/challenge_pointnav2021.local.rgbd.yaml ?

thanks

PPO model evaluation with docker

Hi!

I trained PPO agent model from habitat_baselines folder on pointnav task with gibson dataset. I wanted to evaluate trained model locally using docker. I created the following ppo agent submission script:

import argparse
import habitat
import random
import numpy
import os

from habitat.config import Config
from habitat.config.default import get_config
from habitat_baselines.agents.ppo_agents import PPOAgent


def get_default_config():
    c = Config()
    c.INPUT_TYPE = "rgbd" 
    c.MODEL_PATH = "models/ckpt.199.pth"  # my trained model 
    c.RESOLUTION = 256
    c.HIDDEN_SIZE = 512
    c.RANDOM_SEED = 7
    c.PTH_GPU_ID = 1
    c.GOAL_SENSOR_UUID = "pointgoal"
    return c


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--evaluation", type=str, required=True, choices=["local", "remote"])
    args = parser.parse_args()

    agent_config = get_default_config()
    agent = PPOAgent(agent_config)

    if args.evaluation == "local":
        challenge = habitat.Challenge(eval_remote=False)
    else:
        challenge = habitat.Challenge(eval_remote=True)

    challenge.submit(agent)


if __name__ == "__main__":
    main()

I built the docker and after the command "sudo ./test_locally_pointnav_rgbd.sh --docker-name ppo_submission" got the following error:

2020-03-11 18:50:08,104 Initializing dataset PointNav-v1
2020-03-11 18:50:08,108 initializing sim Sim-v0
2020-03-11 18:50:09,020 Initializing task Nav-v0
Traceback (most recent call last):
  File "ppo_agent.py", line 41, in <module>
    main()
  File "ppo_agent.py", line 37, in main
    challenge.submit(agent)
  File "/habitat-api/habitat/core/challenge.py", line 19, in submit
    metrics = super().evaluate(agent)
  File "/habitat-api/habitat/core/benchmark.py", line 159, in evaluate
    return self.local_evaluate(agent, num_episodes)
  File "/habitat-api/habitat/core/benchmark.py", line 133, in local_evaluate
    action = agent.act(observations)
  File "/habitat-api/habitat_baselines/agents/ppo_agents.py", line 134, in act
    deterministic=False,
  File "/habitat-api/habitat_baselines/rl/ppo/policy.py", line 40, in act
    observations, rnn_hidden_states, prev_actions, masks
  File "/opt/conda/envs/habitat/lib/python3.6/site-packages/torch-1.4.0-py3.6-linux-x86_64.egg/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/habitat-api/habitat_baselines/rl/ppo/policy.py", line 167, in forward
    perception_embed = self.visual_encoder(observations)
  File "/opt/conda/envs/habitat/lib/python3.6/site-packages/torch-1.4.0-py3.6-linux-x86_64.egg/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/habitat-api/habitat_baselines/rl/models/simple_cnn.py", line 147, in forward
    return self.cnn(cnn_input)
  File "/opt/conda/envs/habitat/lib/python3.6/site-packages/torch-1.4.0-py3.6-linux-x86_64.egg/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/envs/habitat/lib/python3.6/site-packages/torch-1.4.0-py3.6-linux-x86_64.egg/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/opt/conda/envs/habitat/lib/python3.6/site-packages/torch-1.4.0-py3.6-linux-x86_64.egg/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/envs/habitat/lib/python3.6/site-packages/torch-1.4.0-py3.6-linux-x86_64.egg/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/opt/conda/envs/habitat/lib/python3.6/site-packages/torch-1.4.0-py3.6-linux-x86_64.egg/torch/nn/functional.py", line 1370, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: size mismatch, m1: [1 x 99712], m2: [25088 x 512] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:290

I am wondering what could cause such error? Please help)

Category list of Task 2

Hi,

As the description said there is a subset of the original MP3D category list that only contains 21 items. I was wondering if it is publicly accessible? If it is, where we can find it?

Thanks!

one question about baseline

I know this challenge is over, but I want to have a try.
I try the script for local evaluation with rgb.pth, I follow the instruction without changes. I find the result I run is 0.4626, but the result from the leaderboard is 0.47. Is that the same model between the repo and the leaderboard?

PPO model training with habitat 2020 challenge config

@mathfac @dhruvbatra Hi!
Another issue on my side I am struggling with is the right training of habitat baseline ppo model with 2020 challenge configuration for pointnav task using habitat-api.

As PPO agent configuration I use the following file ppo_pointnav.yaml

BASE_TASK_CONFIG_PATH: "configs/tasks/pointnav_gib_rgbd_2020.yaml"
TRAINER_NAME: "ppo"
ENV_NAME: "NavRLEnv"
SIMULATOR_GPU_ID: 1
TORCH_GPU_ID: 1

VIDEO_OPTION: ["disk", "tensorboard"]
TENSORBOARD_DIR: "tb"
VIDEO_DIR: "video_dir"
TEST_EPISODE_COUNT: 994
EVAL_CKPT_PATH_DIR: "data/ppo_2020_checkpoints"

NUM_PROCESSES: 4

SENSORS: ["DEPTH_SENSOR"]
CHECKPOINT_FOLDER: "data/ppo_2020_checkpoints"
NUM_UPDATES: 270000
LOG_INTERVAL: 25
CHECKPOINT_INTERVAL: 2000

RL:
  PPO:
    clip_param: 0.1
    ppo_epoch: 4
    num_mini_batch: 2
    value_loss_coef: 0.5
    entropy_coef: 0.01
    lr: 2.5e-4
    eps: 1e-5
    max_grad_norm: 0.5
    num_steps: 128
    hidden_size: 512
    use_gae: True
    gamma: 0.99
    tau: 0.95
    use_linear_clip_decay: True
    use_linear_lr_decay: True
    reward_window_size: 50

And for task configuration I used the same parameters as in challenge_pointnav2020.local.rgbd.yaml file:

ENVIRONMENT:
  MAX_EPISODE_STEPS: 500
SIMULATOR:
  AGENT_0:
    SENSORS: ['RGB_SENSOR', 'DEPTH_SENSOR']
    HEIGHT: 0.88
    RADIUS: 0.18
  HABITAT_SIM_V0:
    GPU_DEVICE_ID: 0
    ALLOW_SLIDING: False
  RGB_SENSOR:
    WIDTH: 640
    HEIGHT: 360
    HFOV: 70
    POSITION: [0, 0.88, 0]
    NOISE_MODEL: "GaussianNoiseModel"
    NOISE_MODEL_KWARGS:
      intensity_constant: 0.1

  DEPTH_SENSOR:
    WIDTH: 640
    HEIGHT: 360
    HFOV: 70
    MIN_DEPTH: 0.1
    MAX_DEPTH: 10.0
    POSITION: [0, 0.88, 0]
    NOISE_MODEL: "RedwoodDepthNoiseModel"

  ACTION_SPACE_CONFIG: 'pyrobotnoisy'
  NOISE_MODEL:
    ROBOT: "LoCoBot"
    CONTROLLER: 'Proportional'
    NOISE_MULTIPLIER: 0.5

TASK:
  TYPE: Nav-v0
  SUCCESS_DISTANCE: 0.36
  SENSORS: ['POINTGOAL_SENSOR']
  POINTGOAL_SENSOR:
    GOAL_FORMAT: POLAR
    DIMENSIONALITY: 2
  GOAL_SENSOR_UUID: pointgoal
  MEASUREMENTS: ['DISTANCE_TO_GOAL', "SUCCESS", 'SPL']
  SUCCESS:
    SUCCESS_DISTANCE: 0.36

Just changed path to the train dataset (Habitat Challenge Data for Gibson (1.5 GB)):

DATASET:
  TYPE: PointNav-v1
  SPLIT: train
  DATA_PATH: data/datasets/pointnav/gibson/v1/{split}/{split}.json.gz

After runnig command python -u habitat_baselines/run.py --exp-config habitat_baselines/config/pointnav/ppo_pointnav.yaml --run-type I got the following error:

---
 The active scene does not contain semantic annotations. 
---
I0325 20:17:08.559100 8915 simulator.py:143] Loaded navmesh data/scene_datasets/gibson/Monson.navmesh
I0325 20:17:08.559392 8915 simulator.py:155] Recomputing navmesh for agent's height 0.88 and radius 0.18.
I0325 20:17:08.567361  8915 PathFinder.cpp:338] Building navmesh with 275x112 cells
I0325 20:17:08.655342  8915 PathFinder.cpp:606] Created navmesh with 137 vertices 61 polygons
I0325 20:17:08.655371  8915 Simulator.cpp:403] reconstruct navmesh successful
2020-03-25 20:17:08,720 Initializing task Nav-v0
2020-03-25 20:17:11,725 agent number of parameters: 52694149
/home/pryhoda/anaconda3/envs/habitat/lib/python3.6/site-packages/torch-1.4.0-py3.6-linux-x86_64.egg/torch/optim/lr_scheduler.py:122: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [3,0,0], thread: [0,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [3,0,0], thread: [1,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [3,0,0], thread: [2,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [3,0,0], thread: [3,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [0,0,0], thread: [0,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [0,0,0], thread: [1,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [0,0,0], thread: [2,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [0,0,0], thread: [3,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [2,0,0], thread: [0,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [2,0,0], thread: [1,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [2,0,0], thread: [2,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [2,0,0], thread: [3,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [1,0,0], thread: [0,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [1,0,0], thread: [1,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [1,0,0], thread: [2,0,0] Assertion `val >= zero` failed.
/pytorch/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::<unnamed>::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [1,0,0], thread: [3,0,0] Assertion `val >= zero` failed.
Traceback (most recent call last):
  File "habitat_baselines/run.py", line 70, in <module>
    main()
  File "habitat_baselines/run.py", line 40, in main
    run_exp(**vars(args))
  File "habitat_baselines/run.py", line 64, in run_exp
    trainer.train()
  File "/home/pryhoda/HabitatProject/habitat-api/habitat_baselines/rl/ppo/ppo_trainer.py", line 346, in train
    rollouts, current_episode_reward, running_episode_stats
  File "/home/pryhoda/HabitatProject/habitat-api/habitat_baselines/rl/ppo/ppo_trainer.py", line 181, in _collect_rollout_step
    outputs = self.envs.step([a[0].item() for a in actions])
  File "/home/pryhoda/HabitatProject/habitat-api/habitat_baselines/rl/ppo/ppo_trainer.py", line 181, in <listcomp>
    outputs = self.envs.step([a[0].item() for a in actions])
RuntimeError: CUDA error: device-side assert triggered
Exception ignored in: <bound method VectorEnv.__del__ of <habitat.core.vector_env.VectorEnv object at 0x7f8dfa79ea58>>
Traceback (most recent call last):
  File "/home/pryhoda/HabitatProject/habitat-api/habitat/core/vector_env.py", line 468, in __del__
    self.close()
  File "/home/pryhoda/HabitatProject/habitat-api/habitat/core/vector_env.py", line 350, in close
    write_fn((CLOSE_COMMAND, None))
  File "/home/pryhoda/anaconda3/envs/habitat/lib/python3.6/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/home/pryhoda/anaconda3/envs/habitat/lib/python3.6/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/home/pryhoda/anaconda3/envs/habitat/lib/python3.6/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

I am wondering if PPO agent is not adapted to train with 2020 challenge config (it run ok for me with 2019 challenge config - pointnav_gibson_rgbd.yaml ) or it is some issues on my side ? Thanks in advance!

Option to terminate running submission

Hi,

Is there an option to terminate a submission currently being executed?
If not, could you manually terminate all my 3 submissions in the test-std track under user name "karkus" or increase the max number of parallel submissions?

I believe the scripts are hanging because of an extra process i launch, and wont exit unless the container is terminated.

thanks!

Failed to build DD-PPO docker

Just wanted to make initial submission with DD-PPO file without own training.

After running command:
docker build . --file Pointnav_DDPPO_Baseline.Dockerfile -t pointnav_submission
get an error:
ADD failed: stat /var/lib/docker/tmp/docker-builder885297584/demo.ckpt.pth: no such file or directory

Where can I find demo.ckpt.pth file for initial submission ?

Evalai submissions are not scheduled

Evalai submissions don't seem to be scheduled anymore. I have submissions to pointnav test-std and minival that have remained in "submitted" status for over a week now

facebookresearch / habitat-challenge Goto Github PK

habitat-challenge's Issues

Local docker evaluation

Remote docker evaluation

Local non-docker evaluation

Recommend Projects

Recommend Topics

Recommend Org