sql-hkr / offline-rl-research Goto Github PK

View Code? Open in Web Editor NEW

0.0 1.0 0.0 58 KB

オフライン強化学習用フレームワーク及びSCQL，SCQL+Dの実装

License: MIT License

Dockerfile 0.37% Python 99.26% Shell 0.37%

offline-rl cql framework scql bc scql-plus-d

offline-rl-research's People

Contributors

Watchers

offline-rl-research's Issues

WandB.logの修正

修正前

wandb.log(
    {
        "q1_pred": q1_pred.mean(),
        "q_target": q_target.mean(),
        "q1_ood_pred": q1_ood_pred.mean(),
        "pesudo_q_target": pesudo_q_target.mean(),
        "actor_q": q_new_actions.mean(),
        "q1_loss": qf1_loss,
        "q2_loss": qf2_loss,
        "policy_loss": policy_loss,
        "vae_loss": vae_loss,
    },
    step=self._current_epoch,
)

修正後

wandb.log(
    {
        "q1_pred": q1_pred.mean(),
        "q_target": q_target.mean(),
        "q1_ood_pred": q1_ood_pred.mean(),
        "pesudo_q_target": pesudo_q_target.mean(),
        "actor_q": q_new_actions.mean(),
        "loss/q1": qf1_loss,
        "loss/q2": qf2_loss,
        "loss/policy": policy_loss,
        "loss/vae": vae_loss,
    },
    step=self._current_epoch,
)

$$
\mathcal{L}Q(\theta_i)
\triangleq
\mathbb{E}{(s,a,s')\sim \mathcal{D},a'\sim \pi_\phi(\cdot \mid s')}
\left[
\frac{1}{2}
\left(
y-Q_{\theta_i}(s,a)
\right)^2\right]
+
\mathbb{E}{s\sim \mathcal{D},a\sim \pi\phi(\cdot \mid s)}
\left[
Q_{\theta_i}(s,a)
\right]
$$