Inconsistent scores between Local and Remote minival for PointNav about habitat-challenge HOT 5 CLOSED

Hi @abhiskk

The random seed is already being set via the construct_envs() function. So I'm not sure why the randomness still persists. In any case, the variance could also be due to the specific model that I am using. I was mainly concerned about the difference between local and remote docker evaluations. The remote evaluation and local non-docker evaluation seem to be reasonably consistent for me right now. So I hope this should not be a problem. I will close this for now. Thanks!

from habitat-challenge.

Hi @srama2512

• I would expect the score across multiple trials to be consistent for your local non-docker evaluation. Can you add in a env.seed(123) to your code and see if you get identical results across different trials.

• There will be variation of scores for your local docker evaluation compared to remote docker evaluation, this is because the remote docker evaluation runs in a different architecture compared to local docker evaluation. That being said I do expect the results across different runs of remote evaluation to be identical. For local docker evaluation can you seed the env inside habitat/core/benchmark.py, doing that will give you identical score across different trials for local docker evaluation.

from habitat-challenge.

I was wondering if anyone figured out why there may be significant differences between local and remote docker evaluations. A reduction is success rate of ~50% is quite a lot. I wonder if that may be related to the differences in the minival and test split results? 60% success on minival vs. 20% success on test seems quite drastic. Is that to be expected?

from habitat-challenge.

Mini-val is very small (around 30 episodes) and was in no way constructed to be a representative sampling of val episodes (@mathfac do you remember how it was constructed?). Mini-val is there to catch if anything is terribly off between local and remote (and it has done that job! We have caught some things thanks to it), not to gauge performance.

If you run your method on full Gibson val locally, does that more closely match test?

from habitat-challenge.

I thought that minival would have 213 episodes. In any case, I'm getting success rates close to 0.5 with SPL close to 0.3 on the full validation set. This was consistent with different subsets of the validation data that I am using as well. So it does not closely match test. Is it possible that there are some configuration / episode generation inconsistencies between val and test?

from habitat-challenge.