oolongqian / gymnasium-robotics Goto Github PK

This library contains a collection of Reinforcement Learning robotic environments that use the Gymansium API. The environments run with the MuJoCo physics engine and the maintained mujoco python bindings.

The documentation website is at robotics.farama.org, and we have a public discord server (which we also use to coordinate development work) that you can join here: https://discord.gg/YymmHrvS

To install the Gymnasium-Robotics environments use pip install gymnasium-robotics

These environments also require the MuJoCo engine from Deepmind to be installed. Instructions to install the physics engine can be found at the MuJoCo website and the MuJoCo Github repository.

Note that the latest environment versions use the latest mujoco python bindings maintained by the MuJoCo team. If you wish to use the old versions of the environments that depend on mujoco-py, please install this library with pip install gymnasium-robotics[mujoco-py]

We support and test for Python 3.7, 3.8, 3.9, 3.10 and 3.11 on Linux and macOS. We will accept PRs related to Windows, but do not officially support it.

Gymnasium-Robotics includes the following groups of environments:

• Fetch - A collection of environments with a 7-DoF robot arm that has to perform manipulation tasks such as Reach, Push, Slide or Pick and Place.
• Shadow Dexterous Hand - A collection of environments with a 24-DoF anthropomorphic robotic hand that has to perform object manipulation tasks with a cube, egg-object, or pen.
• Shadow Hand Dexterous with Touch Sensors - Variations of the Shadow Dexterous Hand environments that include data from 92 touch sensors in the observation space.

The robotic environments use an extension of the core Gymansium API by inheriting from GoalEnv class. The new API forces the environments to have a dictionary observation space that contains 3 keys:

• observation - The actual observation of the environment
• desired_goal - The goal that the agent has to achieved
• achieved_goal - The goal that the agent has currently achieved instead. The objective of the environments is for this value to be close to desired_goal

This API also exposes the function of the reward, as well as the terminated and truncated signals to re-compute their values with different goals. This functionality is useful for algorithms that use Hindsight Experience Replay (HER).

The following example demonstrates how the exposed reward, terminated, and truncated functions can be used to re-compute the values with substituted goals. The info dictionary can be used to store additional information that may be necessary to re-compute the reward, but that is independent of the goal, e.g. state derived from the simulation.

import gymnasium as gym

env = gym.make("FetchReach-v2")
env.reset()
obs, reward, terminated, truncated, info = env.step(env.action_space.sample())

# The following always has to hold:
assert reward == env.compute_reward(obs["achieved_goal"], obs["desired_goal"], info)
assert truncated == env.compute_truncated(obs["achieved_goal"], obs["desired_goal"], info)
assert terminated == env.compute_terminated(obs["achieved_goal"], obs["desired_goal"], info)

# However goals can also be substituted:
substitute_goal = obs["achieved_goal"].copy()
substitute_reward = env.compute_reward(obs["achieved_goal"], substitute_goal, info)
substitute_terminated = env.compute_terminated(obs["achieved_goal"], substitute_goal, info)
substitute_truncated = env.compute_truncated(obs["achieved_goal"], substitute_goal, info)

The GoalEnv class can also be used for custom environments.

If using the Fetch or Shadow Hand environments, please cite:

@misc{1802.09464,
  Author = {Matthias Plappert and Marcin Andrychowicz and Alex Ray and Bob McGrew and Bowen Baker and Glenn Powell and Jonas Schneider and Josh Tobin and Maciek Chociej and Peter Welinder and Vikash Kumar and Wojciech Zaremba},
  Title = {Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research},
  Year = {2018},
  Eprint = {arXiv:1802.09464},
}

To cite the Shadow Dexterous Hand with Touch Sensors environments, please use:

@article{melnik2021using,
  title={Using tactile sensing to improve the sample efficiency and performance of deep deterministic policy gradients for simulated in-hand manipulation tasks},
  author={Melnik, Andrew and Lach, Luca and Plappert, Matthias and Korthals, Timo and Haschke, Robert and Ritter, Helge},
  journal={Frontiers in Robotics and AI},
  pages={57},
  year={2021},
  publisher={Frontiers}
}