Performance on Hopper-v2 about rlkit HOT 11 CLOSED

Looks like the problem was that the refactored v0.2 code was missing the future entropy term. See #43. Closed with 99e080f.

In particular, here's the hopper plot that I got:

from rlkit.

Thank you for the speedy reply! I ran v0.2 with the default hyper-parameters. I’ll double check that the hyper-parameters I ran with match what you posted.

To confirm, the performance metric is logged to “evaluation/Returns Mean”, right?

Also, would you be so kind to share your plotting code? Did you have to do smoothing to get the solid blue line in your graph? If I plot “evaluation/Return Mean” averaged over 5 seeds directly, I got the hyper zigzag pattern in my graph.

from rlkit.

I'll run it again just to check. Yes, that's the correct metric. I used viskit for plotting and did temporal smoothing. I think the important thing is that the thick, shaded region is about the same width as yours.

from rlkit.

What hyperparameters did you run exactly? I got this after running over 5 seeds.

Also, are you using the latest code? I pushed v0.2 only 3 days ago, which includes a few changes that seem to have helped.

Some relevant hyperparams:

  "batch_size": 256,
  "layer_size": 256,
  "max_path_length": 1000,
  "min_num_steps_before_training": 1000,
  "num_epochs": 3000,
  "num_eval_steps_per_epoch": 5000,
  "num_expl_steps_per_train_loop": 1000,
  "num_trains_per_train_loop": 1000,
  "replay_buffer_size": 1000000,
  "discount": 0.99,
  "policy_lr": 0.0003,
  "qf_lr": 0.0003,
  "reward_scale": 1,
  "soft_target_tau": 0.005,
  "target_update_period": 1,
  "use_automatic_entropy_tuning": true

from rlkit.

Ah, I think the issue is that the paths are sometimes not exactly 1000, e.g. if the agent terminates early. This biases the returns to look worse than they actually are, since it might average in a path that had only length 1 (and therefore a return of ~3). For example, by looking at the average rewards, we see that they're basically the same:

So, this seems like a bug in the logging/eval code, but not in the training (phew!). I'll push a fix soon.

from rlkit.

Thanks! I'll rerun the code and let you know how it goes.

from rlkit.

I'm getting the following now:

Can you smooth out the tensorflow results to see if they results are that different?

from rlkit.

It still looks worse than the tensorflow results unfortunately, especially near the end of training.

from rlkit.

Yeah, it's a bit different... It's not a big difference, but I'll look into it. The only difference I can think of is that I switched to batch training rather than online training, and I'm expecting to add support for online-mode soon.

from rlkit.

Okie, thanks so much!

from rlkit.

What hyperparameters did you run exactly? I got this after running over 5 seeds.

Also, are you using the latest code? I pushed v0.2 only 3 days ago, which includes a few changes that seem to have helped.

Some relevant hyperparams:

  "batch_size": 256,
  "layer_size": 256,
  "max_path_length": 1000,
  "min_num_steps_before_training": 1000,
  "num_epochs": 3000,
  "num_eval_steps_per_epoch": 5000,
  "num_expl_steps_per_train_loop": 1000,
  "num_trains_per_train_loop": 1000,
  "replay_buffer_size": 1000000,
  "discount": 0.99,
  "policy_lr": 0.0003,
  "qf_lr": 0.0003,
  "reward_scale": 1,
  "soft_target_tau": 0.005,
  "target_update_period": 1,
  "use_automatic_entropy_tuning": true

Hi, I was wondering how to change the random seeds?

from rlkit.