Comments (11)
We did test this issue a number of times before launching. There should be no difference between local and remote eval in ObjectNav, but there may be a difference in PointNav because of stochasticity (sampling actuations from the noise model).
Is this PointNav or ObjectNav?
from habitat-challenge.
It's on objectnav. Maybe what it might be is that actions are still non-deterministic?
from habitat-challenge.
@abhiskk, do we need any code to set random seeds similar to eval servers?
from habitat-challenge.
@joel99 can you try submitting again, there was a bug in the objectnav evaluation. Also do rebuild the docker from scratch before you resubmit, run a docker system prune -a
before rebuilding.
from habitat-challenge.
I thought this might be related. I was trying to replicate results across runs in PointNav. I had to disable all the sensor noises (RGB, depth and actuation) to achieve consistent results across trials. Below are the numbers I get for 4 episodes across 3 trials after removing the sensor noises. Is there a way to fix the random seeds for the sensors?
Trial 1:
=========> Episode done --- SPL: 0.000, SoftSPL: 0.153, SR: 0.000, D2G: 4.722, Time per episode: 0.825 mins, ETA: 174.956 mins
=========> Episode done --- SPL: 0.630, SoftSPL: 0.623, SR: 1.000, D2G: 0.035, Time per episode: 0.455 mins, ETA: 95.997 mins
=========> Episode done --- SPL: 0.000, SoftSPL: 0.552, SR: 0.000, D2G: 1.185, Time per episode: 0.356 mins, ETA: 74.661 mins
=========> Episode done --- SPL: 0.000, SoftSPL: 0.690, SR: 0.000, D2G: 2.134, Time per episode: 0.301 mins, ETA: 62.952 mins
Trial 2:
=========> Episode done --- SPL: 0.000, SoftSPL: 0.153, SR: 0.000, D2G: 4.722, Time per episode: 0.824 mins, ETA: 174.655 mins
=========> Episode done --- SPL: 0.630, SoftSPL: 0.623, SR: 1.000, D2G: 0.035, Time per episode: 0.456 mins, ETA: 96.205 mins
=========> Episode done --- SPL: 0.000, SoftSPL: 0.552, SR: 0.000, D2G: 1.185, Time per episode: 0.358 mins, ETA: 75.172 mins
=========> Episode done --- SPL: 0.000, SoftSPL: 0.690, SR: 0.000, D2G: 2.134, Time per episode: 0.305 mins, ETA: 63.671 mins
Trial 3:
=========> Episode done --- SPL: 0.000, SoftSPL: 0.153, SR: 0.000, D2G: 4.722, Time per episode: 0.826 mins, ETA: 175.167 mins
=========> Episode done --- SPL: 0.630, SoftSPL: 0.623, SR: 1.000, D2G: 0.035, Time per episode: 0.456 mins, ETA: 96.221 mins
=========> Episode done --- SPL: 0.000, SoftSPL: 0.552, SR: 0.000, D2G: 1.185, Time per episode: 0.357 mins, ETA: 74.931 mins
=========> Episode done --- SPL: 0.000, SoftSPL: 0.690, SR: 0.000, D2G: 2.134, Time per episode: 0.308 mins, ETA: 64.311 mins
from habitat-challenge.
@srama2512, yes noise for sensors is using numba random seed optimization. Latest habitat-api master has changes to set it:
https://github.com/facebookresearch/habitat-api/blob/c3d52b15c83efbdfb0dd3734e2a90d050ba53c84/habitat/core/env.py#L272-L275
Otherwise you can add next method and call it:
@numba.njit
def _seed_numba(seed: int):
random.seed(seed)
np.random.seed(seed)
from habitat-challenge.
And for context, your issue is unrelated to this discussion (which we believe only affects ObjectNav). Your issue is a direct consequence of having noise models (sensor noise and actuation noise).
from habitat-challenge.
@mathfac - great, I must have missed the latest changes. Thanks!
@dhruvbatra - I raised it here since you had mentioned that
"but there may be a difference in PointNav because of stochasticity (sampling actuations from the noise model)."
Should we expect differences in the local and remote results if the random seeds are set correctly?
from habitat-challenge.
I think @abhiskk can be the best person to fully answer this question.
from habitat-challenge.
Hi,
I am getting very different numbers between local and remote evaluation for objectnav, was the bug resolved?
I did "docker system prune" before building my dockerfile for local evaluation.
from habitat-challenge.
I ran the following command and rebuilt the docker and now I am getting consistent numbers.
docker pull fairembodied/habitat-challenge:testing_2020_habitat_base_docker
I am not sure how the remote eval works, but maybe the docker image tag can be changed every time the base docker image is updated to avoid inconsistencies.
from habitat-challenge.
Related Issues (20)
- distance_to_goal calculation results in infinity, pitch control is not working effectively in continuous action space HOT 5
- Custom Env class HOT 1
- Can not run pretrained weights of ObjectNav HOT 3
- Allow sliding HOT 1
- Settings for 2023
- submission killed by server
- Error when try to local evaluate : "6. Evaluate your docker container locally" HOT 2
- Could not submit a docker at EvalAI page HOT 4
- challenge 2023 HOT 1
- Inapproriate Settings about Continuous Action Space in habitat-chllenge 2023
- Eval.ai submission is not evaluating HOT 2
- deadline of challenge 2023
- miscalculated the deadline
- wrong deadline setting on hosts of challenge 2023!!!
- wrong deadline setting on hosts of challenge 2023!!!
- wrong deadline setting on hosts of challenge 2023!!! HOT 10
- The final list of the challenge phase HOT 3
- ConnectionResetError: [Errno 104] Connection reset by peer HOT 1
- AssertionError: Object attributes not uniquely matched to shortened handle. HOT 1
- AttributeError: Can't get attribute 'VectorEnvFactoryConfig'
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from habitat-challenge.