youngwoon / robot-learning Goto Github PK

Match the output dimension of critic networks. For example, [128, 1] in PPO and [128] in Dreamer). We can reduce the last dimension.

Hello, I was running SAC and noticed that the performance was excellent until around 500,000 steps. After reaching this point, there seemed to be a noticeable downturn. Could you help me understand what might be causing this？ If you could answer my question, I would be very grateful.

In line 282 of ppo_agent.py, the critic is trained using:

value_loss = self._config.value_loss_coeff * (ret - value_pred).pow(2).mean()

where ret is computed as ret = adv + vpred[:-1].

This way of calculating return actually gives q_loss but the critic actually predicts v as here.

It seems like the critic is trained using q-loss but it's used to predict only state values. Could you clarify on this?

Thanks

First of all, thanks for your work. I meet some trouble using your master-branch.
I type the following command, and the errror is attempted relative import with no known parent package

python -m run --run_prefix test --algo ppo --env "Hopper-v2"

I have searched online but they don't work.
Could you please help me?
I just clone your code and install the requirements.txt on Ubuntu 16.04.
I have tested the python 3.6 and 3.7.

In rollout.py, run and run_episode share most of the code. We can merge them and replace run_episode with run(every_episodes=1, is_train=False).