dhruvramani / transformers-rl Goto Github PK

A question about the function update_memory in layers.py line269
May I ask, is there any difference between writing this way by cat and appending the hidden_states directly to the new_memory?
Maybe it’s a detail somewhere I didn’t notice,Thanks.

with torch.no_grad():
    new_memory = []
    end_idx = mem_len + seq_len
    beg_idx = max(0, end_idx - mem_len)
    for m, h in zip(previous_memory, hidden_states):
        cat = torch.cat([m, h], dim=0)
        new_memory.append(cat[beg_idx:end_idx].detach())

with torch.no_grad():
    new_memory = []
    for h in hidden_states:
        new_memory.append(h.detach())

If my observation is an image of shape (4, 84, 84), and action dim is 3, so how to modify the code below?

if __name__ == '__main__':
    states = torch.randn(1,1, 4) # seq_size, batch_size, dim - better if dim % 2 == 0
    print("=> Testing Policy")
    policy = TransformerGaussianPolicy(state_dim=states.shape[-1], act_dim=4)
    for i in range(10):
        act = policy(states)
        action = act[0].sample()
        print(torch.isnan(action).any(), action.shape)

At first, thanks for the great implementation.

I found that this initialization is somewhat critical:

It may occur that self.u and self.v will be initialized containing nan. Eventually, this leads to everything becoming nan.

I found 2 bugs in transformer-xl code layers.py.

1. The init_mem function uses an incorrect shape.
   

   Transformers-RL/layers.py

   Lines 261 to 268 in 337d84a

   	def init_memory(self, device=torch.device("cpu")):
   	return [
   	# torch.empty(0, dtype=torch.float).to(device)
   	torch.zeros(20, 5, 8, dtype=torch.float).to(device)
   	for _ in range(self.n_layers + 1)
   	]

def init_memory(self, device=torch.device("cpu")): 
     return [ 
         torch.empty(0, dtype=torch.float).to(device) 
         for _ in range(self.n_layers + 1) 
     ] 

2. The calculation of the beginning index in update_mem is incorrect.
   

   Transformers-RL/layers.py

   Lines 280 to 288 in 337d84a

   	with torch.no_grad():
   	new_memory = []
   	end_idx = mem_len + seq_len
   	beg_idx = max(0, end_idx - mem_len)
   	for m, h in zip(previous_memory, hidden_states):
   	cat = torch.cat([m, h], dim=0)
   	new_memory.append(cat[beg_idx:end_idx].detach())
   	return new_memory

            new_memory = []
            end_idx = mem_len + seq_len
            # self.mem_len is the length of memory retention length. It is different with mem_len.
            beg_idx = max(0, end_idx - self.mem_len)

After fixing these bugs above, the memory mechanism still caused incorrect values. I compared the output of the transformer with and without the memory mechanism, and they are totally different.
I tried another stable-transformer code from this repo. If anyone wants to fix this further, he can refer to this code.