LSTM use in PPO about nerorl HOT 7 CLOSED

Hi @npitsillos
the data is fed to the recurrent layer like you said (10, 128, state_dim). The result of the recurrent layer is reshaped to (10 * 128, state_dim). Basically the remaining layers of the model treat each item of the sequence individually. So does the loss, except that padded items are masked. The only component that does utilize 3 dimensions, is the recurrent layer. For the loss and everything else it is just 2 dimensions.

Is this helpful to you?

from nerorl.

You can also check out my baseline implementation, which isolates the recurrent PPO setup as much as possible.
https://github.com/MarcoMeter/recurrent-ppo-truncated-bptt

from nerorl.

Yes this is helpful! And also I was referencing that repo before in the issue the code was the same and I got confused as to which repo I should use. One final thing the idea behind the masks is to mask out those timesteps that have been padded with 0s?

Thank you again for your help!

from nerorl.

As a final example the mask is supposed to be of the same dimensionality (10,128) to select those values for the loss

from nerorl.

Yes, you are right. The mask masks out time steps that were added due to zero padding.
The masks' shape is (10*128). As it is not part of the forward pass of the model it does not need to be in the shape of (sequence length, num sequences.

from nerorl.

Thank you very much for the explanation! This has helped me a lot in implementing my own repo!

from nerorl.

No problem, you are welcome!

from nerorl.