Repository for the paper "Exciting Action: Investigating Efficient Exploration for Learning Musculoskeletal Humanoid Locomotion".

Abstract: Learning a locomotion controller for a muscu- loskeletal system is challenging due to over-actuation and high- dimensional action space. While many reinforcement learning methods attempt to address this issue, they often struggle to learn human-like gaits because of the complexity involved in engineering an effective reward function. In this paper, we demonstrate that adversarial imitation learning can address this issue by analyzing key problems and providing solutions using both current literature and novel techniques. We vali- date our methodology by learning walking and running gaits on a simulated humanoid model with 16 degrees of free- dom and 92 Muscle-Tendon Units, achieving natural-looking gaits with only a few demonstrations.

In this repository we strictly separate between the code for the policies and objectives described in the paper and the experiments that work with them:

• The code for all experiments can be found here.
• Implementation of the squashed gaussian policy.
• Implementation of the beta distribution policy with different types of parameterization.
• Implementation of Latent Exploration.
• Implementation of SAR as adapted from here.
• Implementation of the Uniform KL Divergence Objective.

1. Create a new conda environment and activate it:

conda create --name musculoco_paper python=3.8
conda activate musculoco_paper

2. Install the Experiment Launcher from source.

3. Install the imitation_lib from source. It provides the GAIL implementation used throughout this work.

4. Clone the LocoMuJoCo repository and install the muscle_act_obs branch from source. Using this specific branch will not be necessary in later versions.

5. Download the motion capture datasets with the following command. This will take up about 4.5 GB of disc space.

loco-mujoco-download-real

6. Install torch:

pip install torch

7. Install the code as a python package by running the following in the top directory of this repository.

pip install -e .

Now with your conda environment activated, you can run any experiment by cd-ing into its directory and running for instance:

python launcher_walk.py

Note: The amount of steps_per_epoch currently set for local running is just for testing the code. All experiments were run with 100.000 steps_per_epoch.

• TBD