Would this library be usable to do a real time deep learning for a furuta pendulum? about rl-tools HOT 2 OPEN

So I decided to use this library and I am currently writing my own vanilla policy gradient. It seems that the td3 algorithm has the assumption that the state of the environment doesn't automatically get changed due to an observation/action, but with my plan for my project I was going to have it make an observation, and then make an action and get a reward approximately one fixed eval period later, so I figured I couldn't have the two critics making observations and getting rewards as the state would have changed.

from rl-tools.

Hi @DaxLynch sorry for the late reply. That sounds like a very cool project! Since TD3 is an off-policy method you can populate the replay buffer however you like. I would suggest to create your own training loop by copying the operations from here:

But removing the step(device, ts.off_policy_runner, ts.actor_critic.actor, ts.actor_buffers_eval, ts.rng); where the off-policy runner is used to generate samples from the environment and place them into the replay buffer.

Then you can have your own code using the policy to generate actions (like e.g. in the evaulation code:

It is probably best to do a full rollout of the trajectory and then, when the robot is idle, do some steps of training to prevent the training to interfere with the control frequency.

I think this is a common use-case (where the interaction/inference can not be in lock-step with and governed by the training loop) and I'm thinking of ways to better accommodate it. Once I find some time, I'll probably separate out the rollouts of the off-policy runner into a separate loop so that it can be optional and create an example where the rollouts are conducted independently of the training

from rl-tools.