Welcome to the Tonic RL library!
Please take a look at the paper for details and results.
The main design principles are:
-
Modularity: Building blocks for creating RL agents, such as models, replays, or exploration strategies, are implemented as configurable modules.
-
Readability: Agents are written in a simple way with an identical API and logs are nicely displayed on the terminal with a progress bar.
-
Fair comparison: The training pipeline is unique and compatible with all Tonic agents and environments. Agents are defined by their core ideas while general tricks/improvements like non-terminal timeouts, observation normalization and action scaling are shared.
-
Benchmarking: Benchmark data of the provided agents trained on 70 continuous control environments are provided for direct comparison.
-
Wrapped popular environments: Environments from OpenAI Gym, PyBullet and DeepMind Control Suite are made compatible with non-terminal timeouts and synchronous distributed training.
-
Compatibility with different ML frameworks: Both TensorFlow 2 and PyTorch are currently supported. Simply import
tonic.tensorflowortonic.torch. -
Experimenting from the console: While launch scripts can be used, iterating over various configurations from a console is made possible using snippets of Python code directly.
-
Visualization of trained agents: Experiment configurations and checkpoints can be loaded to play with trained agents.
-
Collection of trained models: To keep the main Tonic repository light, the full logs and trained models from the benchmark are available in the tonic_data repository.
Download and install Tonic:
git clone https://github.com/fabiopardo/tonic.git
pip install -e tonic/Install TensorFlow or PyTorch, for example using:
pip install tensorflow torchUse TensorFlow or PyTorch to train an agent, for example using:
python3 -m tonic.train \
--header 'import tonic.torch' \
--agent 'tonic.torch.agents.PPO()' \
--environment 'tonic.environments.Gym("BipedalWalker-v3")' \
--name PPO-X \
--seed 0Snippets of Python code are used to directly configure the experiment. This is a very powerful feature allowing to configure agents and environments with various arguments or even load custom modules without adding them to the library. For example:
python3 -m tonic.train \
--header "import sys; sys.path.append('path/to/custom'); from custom import CustomAgent" \
--agent "CustomAgent()" \
--environment "tonic.environments.Bullet('AntBulletEnv-v0')" \
--seed 0By default, environments use non-terminal timeouts, which is particularly important for locomotion tasks. But a time feature can be added to the observations to keep the MDP Markovian. See the Time Limits in RL paper for more details. For example:
python3 -m tonic.train \ ⏎
--header 'import tonic.tensorflow' \
--agent 'tonic.tensorflow.agents.PPO()' \
--environment 'tonic.environments.Gym("Reacher-v2", terminal_timeouts=True, time_feature=True)' \
--seed 0Distributed training can be used to accelerate learning. In Tonic, groups of sequential workers can be launched in parallel processes using for example:
python3 -m tonic.train \
--header "import tonic.tensorflow" \
--agent "tonic.tensorflow.agents.PPO()" \
--environment "tonic.environments.Gym('HalfCheetah-v3')" \
--parallel 10 --sequential 100 \
--seed 0During training, the experiment configuration, logs and checkpoints are
saved in environment/agent/seed/.
Result can be plotted with:
python3 -m tonic.plot --path BipedalWalker-v3/ --baselines allRegular expressions like BipedalWalker-v3/PPO-X/0,
BipedalWalker-v3/{PPO*,DDPG*} or *Bullet* can be used to point to different
sets of logs.
The --baselines argument can be used to load logs from the benchmark. For
example --baselines all uses all agents while --baselines A2C PPO TRPO will
use logs from A2C, PPO and TRPO.
Different headers can be used for the x and y axes, for example to compare the
gain in wall clock time of using distributed training, replace --parallel 10
with --parallel 5 in the last training example and plot the result with:
python3 -m tonic.plot --path HalfCheetah-v3/ --x_axis train/seconds --x_label SecondsAfter some training time, checkpoints are generated and can be used to play with the trained agent:
python3 -m tonic.play --path BipedalWalker-v3/PPO-X/0Environments are rendered using the appropriate framework. For example, when
playing with DeepMind Control Suite environments, policies are loaded in a
dm_control.viewer where Space is used to start the interaction, Backspace
is used to start a new episode, [ and ] are used to switch cameras and
double click on a body part followed by Ctrl + mouse clicks is used to add
perturbations.
The tonic_data repository can be downloaded with:
git clone https://github.com/fabiopardo/tonic_data.gitThe best seed for each agent is stored in environment/agent and can be
reloaded using for example:
python3 -m tonic.play --path tonic_data/tensorflow/humanoid-stand/TD3
The full benchmark plots are available here.
They can be generated with:
python3 -m tonic.plot \
--baselines all \
--backend agg --columns 7 --font_size 17 --legend_font_size 30 --legend_marker_size 20 \
--name benchmarkOr:
python3 -m tonic.plot \
--path tonic_data/tensorflow \
--backend agg --columns 7 --font_size 17 --legend_font_size 30 --legend_marker_size 20 \
--name benchmarkAnd a selection can be generated with:
python3 -m tonic.plot \
--path tonic_data/tensorflow/{AntBulletEnv-v0,BipedalWalker-v3,finger-turn_hard,fish-swim,HalfCheetah-v3,HopperBulletEnv-v0,Humanoid-v3,quadruped-walk,swimmer-swimmer15,Walker2d-v3} \
--backend agg --columns 5 --font_size 20 --legend_font_size 30 --legend_marker_size 20 \
--name selection
Tonic was inspired by a number of other deep RL code bases. In particular, OpenAI Baselines, Spinning Up in Deep RL and Acme.
If you use Tonic in your research, please cite the paper:
@article{pardo2020tonic,
title={Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking},
author={Pardo, Fabio},
journal={arXiv preprint arXiv:2011.07537},
year={2020}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent