1. Atari Env for evaluation uses gymnasium, instead of atari_py
2. Support Tensorboard & WandB
3. Better performance than offical coding in Breakout
4. Three models related to Transformer with offline RL: DT, RADT, StARformer (see reference)

Run 3 different seeds for each model, test 10 times at each evaluation. WandB online

Evaluate Breakout by model StARformer training with seed=0, 391 scores.

Evaluate Pong by model StARformer training with seed=0, 20 scores.

Evaluate Boxing by model StARformer training with seed=0, 94 scores.

Evaluate Qbert by model StARformer training with seed=0, 14550 scores.

Evaluate Seaquest by model StARformer training with seed=0, 1480 scores.

Use the same training dataset as GitHub-decision-transformer, download dataset, please refer to readme-atari.md. Since we just need 1% dataset, bellow cmd can download minimal size:

mkdir -p ./dqn_replay/Breakout
gsutil -m cp -R gs://atari-replay-datasets/dqn/Breakout/1 ./dqn_replay/Breakout

You can replace name of environment Breakout with Pong, Assault, Boxing, Qbert, Seaquest

• run.sh for Decision Transformer.
• run_RADT.sh for Return-Aligned Decision Transformer.
• run_StAR.sh for StARformer.
• run_ViDformer.sh for ViDformer.

Change the --path-buffer-root to your dqn_replay path in each running script.

NOTE: DatasetBuilder will create ./dqn_reply/[EnvName]/dt_pkl for saving .pkl cache file after first loading replay buffer, this can accelerate reading for next time.

run_StAR.sh --game [EnvName] \ # Change EnvName in [Breakout, Pong, Assault, Boxing, Qbert, Seaquest]
--wandb  # Upload tensorboard to wandb

1. Decision-Transformer: https://arxiv.org/pdf/2106.01345.pdf, official code: https://github.com/kzl/decision-transformer
2. Return-Aligned Decision Transformer: https://arxiv.org/pdf/2402.03923.pdf
3. StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learning: https://arxiv.org/pdf/2110.06206.pdf, official code: https://github.com/elicassion/StARformer