The Autonomous Learning Library (all
) is an object-oriented deep reinforcement learning library in pytorch
. The goal of the library is to provide implementations of modern reinforcement learning algorithms that reflect the way that reinforcement learning researchers think about agent design and to provide the components necessary to build and test new ideas with minimal overhead.
As of today, all
contains implementations of the following deep RL algorithms:
- Advantage Actor-Critic (A2C)
- Categorical DQN (C51)
- Deep Deterministic Policy Gradient (DDPG)
- Deep Q-Learning (DQN) + extensions
- Proximal Policy Optimization (PPO)
- Soft Actor-Critic (SAC)
It also contains implementations of the following "vanilla" agents, which provide useful baselines and perform better than you may expect:
- Vanilla Actor-Critic
- Vanilla Policy Gradient
- Vanilla Q-Learning
- Vanilla Sarsa
We will try to stay up-to-date with advances in the field, but we do not intend to implement every algorithm. Rather, we prefer to maintain a smaller set of high-quality agents that have achieved notoriety in the field.
The primary reason for using all
over its many competitors is because it contains components that allow you to build your own reinforcement learning agents.
We provide out-of-the-box modules for:
- Custom Q-Networks, V-Networks, policy networks, and feature networks
- Generic function approximation
- Experience Replay
- Prioritized Experience Replay
- Advantage Estimation
- Generalized Advantage Estimation (GAE)
- Target networks
- Polyak averaging
- Easy parameter and learning rate scheduling
- An enhanced
nn
module (includes dueling layers, noisy layers, action bounds, and the covetednn.Flatten
) -
gym
topytorch
wrappers - Atari wrappers
- An
Experiment
API for comparing and evaluating agents - A
SlurmExperiment
API for running massive experiments on computing clusters - A
Writer
object for easily logging information intensorboard
Rather than being embedded in the agents, all of these modules are available for use by your own custom agents.
Additionally, the included agents accept custom versions of any of the above objects.
Have a new type of replay buffer in mind?
Code it up and pass it directly to our DQN
and DDPG
implementations.
Additionally, our agents were written with readibility as a primary concern, so they are easy to modify.
Our agents implement a single method: action = agent.act(state, reward)
.
Much of the complexity is handled behind the scenes, making the final API simple.
Unlike many libraries, we do not combine the learning algorithm and the training loop.
Instead, our agents can be embedded in existing applications and trained in the online setting.
The all.presets
includes agents that preconfigured for certain types of environments.
It can be used as follows:
from all.presets.atari import dqn
from all.environments import AtariEnvironment
env = AtariEnvironment('Breakout')
agent = dqn(lr=3e-4)(env)
while True:
if env.done:
env.reset()
else:
env.step(action)
env.render()
action = agent.act(env.state, env.reward)
However, generally we recommend using the Experiment
API, which adds many additional features:
from all.presets.atari import a2c, dqn
from all.environments import AtariEnvironment
from all.experiments import Experiment
# use graphics card for much faster training
device = 'cuda'
experiment = Experiment(AtariEnvironment('Breakout', device=device), frames=10e6)
experiment.run(a2c(device=device))
experiment.run(dqn(device=device))
Results can be viewed by typing:
make tensorboard
This library is built on top of pytorch
.
If you don't want your trials to take forever, it is highly recommended that you make sure your installation has CUDA support (and that you have a CUDA compatible GPU).
You'll also need tensorflow
in order to use tensorboard
(used for storing and plotting runs).
There are two ways to install the autonomous-learning-library
: a "light" installation, which assumes that the major dependencies are already installed, and a "full" installation which installs everything from scratch.
Use this if you already have pytorch
and tensorflow
installed.
Simply run:
pip install -e .
Presto! If you have any trouble with installing the Gym environments, check out their GitHub page and try whatever they recommend in [current year].
If you're on Linux and don't have pytorch
or tensorflow
installed, we did you the courtesy of providing a helpful install script:
make install
With any luck, the all
library should now be installed!
The unit tests may be run using:
make test
If the unit tests pass with no errors, it is more than likely that your installation works! The unit test run every agent using both cpu
and cuda
for a few timesteps/episodes.
You can easily benchmark the included algorithms using the scripts in ./benchmarks
.
To run a simple CartPole
benchmark, run:
python scripts/classic.py CartPole-v1 dqn
Results are printed to the console, and can also be viewed by running:
make tensorboard
and opening your browser to http://localhost:6006.
To run an Atari benchmark in CUDA mode (warning: this could take several hours to run, depending on your machine):
python scripts/atari.py Pong dqn
If you want to run in cpu
mode (~10x slower on my machine), you can add --device cpu
:
python scipts/atari.py Pong dqn --device cpu
This library was built at the Autonomous Learning Laboratory (ALL) at the University of Massachusetts, Amherst. It was written and is currently maintained by Chris Nota (@cpnota). The views expressed or implied in this repository do not necessarily reflect the views of the ALL.