Comments (5)
Hi @abhiskk
The random seed is already being set via the construct_envs()
function. So I'm not sure why the randomness still persists. In any case, the variance could also be due to the specific model that I am using. I was mainly concerned about the difference between local and remote docker evaluations. The remote evaluation and local non-docker evaluation seem to be reasonably consistent for me right now. So I hope this should not be a problem. I will close this for now. Thanks!
from habitat-challenge.
Hi @srama2512
-
I would expect the score across multiple trials to be consistent for your local non-docker evaluation. Can you add in a
env.seed(123)
to your code and see if you get identical results across different trials. -
There will be variation of scores for your local docker evaluation compared to remote docker evaluation, this is because the remote docker evaluation runs in a different architecture compared to local docker evaluation. That being said I do expect the results across different runs of remote evaluation to be identical. For local docker evaluation can you seed the env inside
habitat/core/benchmark.py
, doing that will give you identical score across different trials for local docker evaluation.
from habitat-challenge.
I was wondering if anyone figured out why there may be significant differences between local and remote docker evaluations. A reduction is success rate of ~50% is quite a lot. I wonder if that may be related to the differences in the minival and test split results? 60% success on minival vs. 20% success on test seems quite drastic. Is that to be expected?
from habitat-challenge.
Mini-val is very small (around 30 episodes) and was in no way constructed to be a representative sampling of val episodes (@mathfac do you remember how it was constructed?). Mini-val is there to catch if anything is terribly off between local and remote (and it has done that job! We have caught some things thanks to it), not to gauge performance.
If you run your method on full Gibson val locally, does that more closely match test?
from habitat-challenge.
I thought that minival would have 213 episodes. In any case, I'm getting success rates close to 0.5 with SPL close to 0.3 on the full validation set. This was consistent with different subsets of the validation data that I am using as well. So it does not closely match test. Is it possible that there are some configuration / episode generation inconsistencies between val and test?
from habitat-challenge.
Related Issues (20)
- distance_to_goal calculation results in infinity, pitch control is not working effectively in continuous action space HOT 5
- Custom Env class HOT 1
- Can not run pretrained weights of ObjectNav HOT 3
- Allow sliding HOT 1
- Settings for 2023
- submission killed by server
- Error when try to local evaluate : "6. Evaluate your docker container locally" HOT 2
- Could not submit a docker at EvalAI page HOT 4
- challenge 2023 HOT 1
- Inapproriate Settings about Continuous Action Space in habitat-chllenge 2023
- Eval.ai submission is not evaluating HOT 2
- deadline of challenge 2023
- miscalculated the deadline
- wrong deadline setting on hosts of challenge 2023!!!
- wrong deadline setting on hosts of challenge 2023!!!
- wrong deadline setting on hosts of challenge 2023!!! HOT 10
- The final list of the challenge phase HOT 3
- ConnectionResetError: [Errno 104] Connection reset by peer HOT 1
- AssertionError: Object attributes not uniquely matched to shortened handle. HOT 1
- AttributeError: Can't get attribute 'VectorEnvFactoryConfig'
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from habitat-challenge.