This project was initiated in the RL course of Fall 2021 at The African Master's in Machine Intelligence (AMMI) as a course-project where we implemented the SAC algorithm (Haarnoja et al.) for continuous control tasks. It is now an open project where we care to design code bases and benchmarks for RL algorithms in order to ease the development of new algorithms. We are designing this repo based on existing repositories as well as original papers to produce better general implementations for a selected set of algorithms.
Algorithms we are re-implementing/plannning to re-implement:
Algorithms | Model | Value | On Policy | MPC | Progress | Reference |
---|---|---|---|---|---|---|
VPG | False | V(GAE) | True | False | π’ | Sutton et al., 1999 |
NPG | False | V(GAE) | True | False | π΄ | Kakade, 2001 |
PPO | False | V(GAE) | True | False | π’ | Schulman et al., 2017 |
SAC | False | 2xQ | False | False | π’ | Haarnoja et al., 2018 |
PETS | True | None | None | True | π΄ | Chua et al., 2018 |
MB-PPO | True | V(GAE) | True | False | π’ | Similar~Rajeswaran et al., 2020 |
MB-SAC | True | 2xQ | False | False | π’ | Janner et al., 2019 |
MOVOQ | True | N/A | N/A | N/A | π‘ | N/A |
MoPAC | True | 2xQ | False | True | π£ | Morgan et al., 2021 |
MPC-SAC | True | V(GAE)/2xQ | False | True | π΄ | Omer et al., 2021 |
π’ Done || π‘ Now || π£ Next || π΄ No plan
We aim to finetune our implementations to work with a generalized set of hyperparametrs across different algorithms. We are working with the following hyperparameters in the mean time:
βοΈ | Network | Arch | Act | LRate | MFOV | MFOQ | MBOV | MBOQ | Notes |
---|---|---|---|---|---|---|---|---|---|
Policy | [2x128] | Tanh | 3e-4 | π© | π¨ | π© | π₯ | ||
Policy | [2x256] | ReLU | 3e-4 | π₯ | π© | β¬οΈ | π© | ||
β | Policy | [2x256] | PReLU | 3e-4 | π© | π© | π© | π¦ | |
V | [2x128] | Tanh | 1e-3 | π© | β¬οΈ | π© | β¬οΈ | ||
V | [2x128] | PReLU | 1e-3 | π© | β¬οΈ | π¨ | β¬οΈ | ||
β | V | [2x256] | PReLU | 3e-4 | π© | β¬οΈ | π© | β¬οΈ | |
Q | [2x256] | ReLU | 3e-4 | β¬οΈ | π© | β¬οΈ | π© | ||
β | Q | [2x256] | PReLU | 3e-4 | β¬οΈ | π© | β¬οΈ | π¦ | |
β | V-Model | [2x512] | ReLU | 1e-3 | β¬οΈ | β¬οΈ | π© | π₯ | |
β | Q-Model | [4x200] | Swish | 3e-4 | β¬οΈ | β¬οΈ | π₯ | π© |
π© Best || π¨ Good || π₯ Bad || π¦ In progress
In thoe following we evaluate our code on the following environments. Download gifs from this Google drive folder at drive. Results are averaged across 3 random seeds, and smoothed with 0.75 Exponential Moving Average.
Hopper-v2 | Walker2d-v2 |
---|---|
![]() |
![]() |
HalfCheetah-v2 | Ant-v2 |
![]() |
![]() |
DClaw Valve Turning | ShadowHand Cube Re-orientation |
---|---|
![]() |
![]() |
Move into AMMI-RL/
directory, and then run the following:
conda create -n ammi-rl python=3.8
pip install -e .
pip install numpy torch wandb gym
If you want to run MuJoCo Locomotion tasks, and ShadowHand, you should install MuJoCo first (it's open sourced until 31th Oct), and then install mujoco-py:
sudo apt-get install ffmpeg
pip install -U 'mujoco-py<2.1,>=2.0'
If you are using A local GPU of Nvidia and want to record MuJoCo environments issue link, run:
unset LD_PRELOAD
Move into AMMI-RL/
directory, and then run the following:
conda create -n ammi-rl python=3.8
pip install -e .
pip install numpy torch wandb gym
If you want to run MuJoCo Locomotion tasks, and ShadowHand, you should install MuJoCo first (it's open sourced until 31th Oct), and then install mujoco-py:
brew install ffmpeg gcc
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/.mujoco/mujoco200/bin
pip install -U 'mujoco-py<2.1,>=2.0'
If you are using A local GPU of Nvidia and want to record MuJoCo environments issue link, run:
unset LD_PRELOAD
Move into AMMI-RL/
directory, and then:
python experiment.py -cfg <cfg_file-.py> -seed <int>
for example:
python experiment.py -cfg sac_hopper -seed 1
To evaluate a saved policy model, run the following command:
python evaluate_agent.py -env <env_name> -alg <alg_name> -seed <int> -EE <int>
for example:
python evaluate_agent.py -env Walker2d-v2 -alg SAC -seed 1 -EE 5
(last name alphabetical order) | contribution
- Rami Ahmed | VPG, PPO, SAC, MB{PPO, SAC}
- Wafaa Mohammed | SAC
- Ruba Mutasim | SAC
- MohammedElfatih Salah | SAC
- Bilal Piot, Corentin Tallec and Florian Strub (During RL Course Fall 2021)
- Vlad Mnih, Eszter VΓ©rtes and Theophane Weber (During Rami's AMMI project)
This repo was inspired by many great repos, mostly the following ones (not necessarily in order):