[BUG] Handling heterogeneous environment observations about rl HOT 3 CLOSED

I see the VMAS errors above have been fixed in the nightly package. However, advice would still be appreciated for observation spaces that vary on each step. Judging by #1463, I'm guessing padding the observation is the only way.

from rl.

Thanks for the comments! Here is my take on this.

The heterogeneity discussed in the comments you linked was within a single timestep. Heterogeneity through time is a little more tricky and I think the choice of not allowing it is deliberate. @vmoens maybe can say a few things about this

I think right now the only way to have an observation space with a shape that changes through time is using padding. I.e. you create the space of size Max_N x K. You can pad the non available observations with 0s or any other value of your choice. You can also output a mask among the obs, which tells what obs in the set are valid.

This is what we do for actions spaces with available actions changing through time. There, a mask is output by the env to make the specs know what actions are to be considered available. A similar thing could be done for obs.

This is similar to what we go in PettingZoo when there is a varying number of agents each step or the game is turn-based. You can check out the PettingZoo wrapper for more info on this.

from rl.

Thanks for the comments! Here is my take on this.

The heterogeneity discussed in the comments you linked was within a single timestep. Heterogeneity through time is a little more tricky and I think the choice of not allowing it is deliberate. @vmoens maybe can say a few things about this

I think right now the only way to have an observation space with a shape that changes through time is using padding. I.e. you create the space of size Max_N x K. You can pad the non available observations with 0s or any other value of your choice. You can also output a mask among the obs, which tells what obs in the set are valid.

This is what we do for actions spaces with available actions changing through time. There, a mask is output by the env to make the specs know what actions are to be considered available. A similar thing could be done for obs.

This is similar to what we go in PettingZoo when there is a varying number of agents each step or the game is turn-based. You can check out the PettingZoo wrapper for more info on this.

Thank you! That's what I figured. This isn't a major problem for my application, so I'll close the issue now.

from rl.