For the final project, we are going to investigate one application of Transfer Learning for Reinforcement Learning Task. The OpenAI Gym provides a set of Atari games environments with two input settings: screen frames in the form of RGB images, and RAM (a 256 byte array containing the state of the game). There is no direct way to transform one input to another, since RAM potentially contains state information that is not represented in the pixels. We are interested in developing methods that transfer the policies learned from one input setting to the same game with different input setting. Essentially we want to model a mapping between the two settings such that transferring learned knowledge is possible. We will be using the openAI gym to get the Atari games (https://gym.openai.com/). One game we are considering trying transfer in is Amidar v0 to Amidar - ram v0.
-
Generate matching pairs of RGB images and RAM states.
-
Design and train a neural network (RGB2RAM) to map RGB images to RAM states.
-
Train an RL model (RL-RAM) using RAM inputs.
-
Train an RL model (RL-RGB) using RGB inputs (serves as baseline).
-
Transfer the RL-RAM model on the same task with RGB inputs by first converting the RGB inputs using RGB2RAM, then applying the RL-RAM model.
- compare training time & accuracies with the RL-RGB baseline.
-
(Optional) Fine-tuning to improve accuracy.
-
(Extention) Repeat 1 & 2 for several different games so that RGB2RAM is a general mapping between RGB images and RAM states. Then if there is an Atari game that we only have access to the environment with RGB inputs, while someone else has trained an RL-RAM model, we can transfer the RL-RAM model to our environment even we don't have the RAM inputs.
(This can be apply to a broader case, where someone else has trained and published a model, but you don't have access to the same inputs they use to train the model; you do have access to similar games where you have both inputs, so you can learn a mapping between the two inputs to make use of the already-trained model.)
-
Collect data for RGB2RAM.
save_atari_observations.py
defines an dqn agent that records both RGB and RAM in callbacks during training.-
Run
python examples/save_atari_observations.py [--game Breakout]
to train a model on RGB and save observations. -
To specify the frequency of saving observations, run with options:
python examples/save_atari_observations.py --save_every_episode 10 --save_every_step 20
-
Observations are saved under
./train_history/
directory.
-
-
Train RGB2RAM.
-
Go to
./rgb2ram
directory, runpython3 utils.py
to get training data from./train_history/
and save to./rgb2ram/data
in ".npz" format. -
Under
./rgb2ram
, to learn mapping between RGB and RAM, run:python3 train.py [--game_name Bowling --model CNN2]
Possible model options are: 'FF', 'CNN1', 'CNN2', 'LSTM'.
-
-
Run
python examples/new_dqn_atari.py --help
to see how to use transfer learning models to train RL agent.eg.
python3 examples/new_dqn_atari.py --mode transfer --game_name Breakout --transfer_model cnn2 --rl_agent big_ram
-
Train a model on Ram.
-
Train RGB2RAM model:
- try different architectures
- try different loss and optimizer
- try different dataset sizes
![]() |
![]() |
![]() |
keras-rl
implements some state-of-the art deep reinforcement learning algorithms in Python and seamlessly integrates with the deep learning library Keras.
Furthermore, keras-rl
works with OpenAI Gym out of the box. This means that evaluating and playing around with different algorithms is easy.
Of course you can extend keras-rl
according to your own needs. You can use built-in Keras callbacks and metrics or define your own.
Even more so, it is easy to implement your own environments and even algorithms by simply extending some simple abstract classes. Documentation is available online.
As of today, the following algorithms have been implemented:
- Deep Q Learning (DQN) [1], [2]
- Double DQN [3]
- Deep Deterministic Policy Gradient (DDPG) [4]
- Continuous DQN (CDQN or NAF) [6]
- Cross-Entropy Method (CEM) [7], [8]
- Dueling network DQN (Dueling DQN) [9]
- Deep SARSA [10]
- Asynchronous Advantage Actor-Critic (A3C) [5]
- Proximal Policy Optimization Algorithms (PPO) [11]
You can find more information on each agent in the doc.
- Install Keras-RL from Pypi (recommended):
pip install keras-rl
- Install from Github source:
git clone https://github.com/keras-rl/keras-rl.git
cd keras-rl
python setup.py install
If you want to run the examples, you'll also have to install:
- gym by OpenAI: Installation instruction
- h5py: simply run
pip install h5py
For atari example you will also need:
- Pillow:
pip install Pillow
- gym[atari]: Atari module for gym. Use
pip install gym[atari]
Once you have installed everything, you can try out a simple example:
python examples/dqn_cartpole.py
This is a very simple example and it should converge relatively quickly, so it's a great way to get started! It also visualizes the game during training, so you can watch it learn. How cool is that?
Some sample weights are available on keras-rl-weights.
If you have questions or problems, please file an issue or, even better, fix the problem yourself and submit a pull request!
You're using Keras-RL on a project? Open a PR and share it!
To see graphs of your training progress and compare across runs, run pip install wandb
and add the WandbLogger callback to your agent's fit()
call:
from rl.callbacks import WandbLogger
...
agent.fit(env, nb_steps=50000, callbacks=[WandbLogger()])
For more info and options, see the W&B docs.
If you use keras-rl
in your research, you can cite it as follows:
@misc{plappert2016kerasrl,
author = {Matthias Plappert},
title = {keras-rl},
year = {2016},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/keras-rl/keras-rl}},
}
- Playing Atari with Deep Reinforcement Learning, Mnih et al., 2013
- Human-level control through deep reinforcement learning, Mnih et al., 2015
- Deep Reinforcement Learning with Double Q-learning, van Hasselt et al., 2015
- Continuous control with deep reinforcement learning, Lillicrap et al., 2015
- Asynchronous Methods for Deep Reinforcement Learning, Mnih et al., 2016
- Continuous Deep Q-Learning with Model-based Acceleration, Gu et al., 2016
- Learning Tetris Using the Noisy Cross-Entropy Method, Szita et al., 2006
- Deep Reinforcement Learning (MLSS lecture notes), Schulman, 2016
- Dueling Network Architectures for Deep Reinforcement Learning, Wang et al., 2016
- Reinforcement learning: An introduction, Sutton and Barto, 2011
- Proximal Policy Optimization Algorithms, Schulman et al., 2017