Inconsistent Scores Between Local and Remote minival for ObjectNav about habitat-challenge HOT 11 CLOSED

We did test this issue a number of times before launching. There should be no difference between local and remote eval in ObjectNav, but there may be a difference in PointNav because of stochasticity (sampling actuations from the noise model).

Is this PointNav or ObjectNav?

from habitat-challenge.

It's on objectnav. Maybe what it might be is that actions are still non-deterministic?

from habitat-challenge.

@abhiskk, do we need any code to set random seeds similar to eval servers?

from habitat-challenge.

@joel99 can you try submitting again, there was a bug in the objectnav evaluation. Also do rebuild the docker from scratch before you resubmit, run a docker system prune -a before rebuilding.

from habitat-challenge.

I thought this might be related. I was trying to replicate results across runs in PointNav. I had to disable all the sensor noises (RGB, depth and actuation) to achieve consistent results across trials. Below are the numbers I get for 4 episodes across 3 trials after removing the sensor noises. Is there a way to fix the random seeds for the sensors?

Trial 1:
=========> Episode done --- SPL: 0.000, SoftSPL: 0.153, SR: 0.000, D2G: 4.722, Time per episode: 0.825 mins, ETA: 174.956 mins
=========> Episode done --- SPL: 0.630, SoftSPL: 0.623, SR: 1.000, D2G: 0.035, Time per episode: 0.455 mins, ETA: 95.997 mins
=========> Episode done --- SPL: 0.000, SoftSPL: 0.552, SR: 0.000, D2G: 1.185, Time per episode: 0.356 mins, ETA: 74.661 mins
=========> Episode done --- SPL: 0.000, SoftSPL: 0.690, SR: 0.000, D2G: 2.134, Time per episode: 0.301 mins, ETA: 62.952 mins

Trial 2:
=========> Episode done --- SPL: 0.000, SoftSPL: 0.153, SR: 0.000, D2G: 4.722, Time per episode: 0.824 mins, ETA: 174.655 mins
=========> Episode done --- SPL: 0.630, SoftSPL: 0.623, SR: 1.000, D2G: 0.035, Time per episode: 0.456 mins, ETA: 96.205 mins
=========> Episode done --- SPL: 0.000, SoftSPL: 0.552, SR: 0.000, D2G: 1.185, Time per episode: 0.358 mins, ETA: 75.172 mins
=========> Episode done --- SPL: 0.000, SoftSPL: 0.690, SR: 0.000, D2G: 2.134, Time per episode: 0.305 mins, ETA: 63.671 mins

Trial 3:
=========> Episode done --- SPL: 0.000, SoftSPL: 0.153, SR: 0.000, D2G: 4.722, Time per episode: 0.826 mins, ETA: 175.167 mins
=========> Episode done --- SPL: 0.630, SoftSPL: 0.623, SR: 1.000, D2G: 0.035, Time per episode: 0.456 mins, ETA: 96.221 mins
=========> Episode done --- SPL: 0.000, SoftSPL: 0.552, SR: 0.000, D2G: 1.185, Time per episode: 0.357 mins, ETA: 74.931 mins
=========> Episode done --- SPL: 0.000, SoftSPL: 0.690, SR: 0.000, D2G: 2.134, Time per episode: 0.308 mins, ETA: 64.311 mins

from habitat-challenge.

@srama2512, yes noise for sensors is using numba random seed optimization. Latest habitat-api master has changes to set it:
https://github.com/facebookresearch/habitat-api/blob/c3d52b15c83efbdfb0dd3734e2a90d050ba53c84/habitat/core/env.py#L272-L275
Otherwise you can add next method and call it:

    @numba.njit
    def _seed_numba(seed: int):
        random.seed(seed)
        np.random.seed(seed)

from habitat-challenge.

And for context, your issue is unrelated to this discussion (which we believe only affects ObjectNav). Your issue is a direct consequence of having noise models (sensor noise and actuation noise).

from habitat-challenge.

@mathfac - great, I must have missed the latest changes. Thanks!
@dhruvbatra - I raised it here since you had mentioned that
"but there may be a difference in PointNav because of stochasticity (sampling actuations from the noise model)."

Should we expect differences in the local and remote results if the random seeds are set correctly?

from habitat-challenge.

I think @abhiskk can be the best person to fully answer this question.

from habitat-challenge.

Hi,
I am getting very different numbers between local and remote evaluation for objectnav, was the bug resolved?

I did "docker system prune" before building my dockerfile for local evaluation.

from habitat-challenge.

I ran the following command and rebuilt the docker and now I am getting consistent numbers.
docker pull fairembodied/habitat-challenge:testing_2020_habitat_base_docker

I am not sure how the remote eval works, but maybe the docker image tag can be changed every time the base docker image is updated to avoid inconsistencies.

from habitat-challenge.