farama-foundation / shimmy Goto Github PK

View Code? Open in Web Editor NEW

120.0 7.0 19.0 28.18 MB

An API conversion tool for popular external reinforcement learning environments

Home Page: https://shimmy.farama.org

License: MIT License

Python 93.32% Shell 1.14% Dockerfile 5.54%

deepmind-control-suite deepmind-lab gym-environment gymnasium reinforcement-learning

shimmy's People

Contributors

Stargazers

Watchers

Forkers

redtachyon pseudo-rnd-thoughts delaschwein vwxyzjn rodrigodelazcano elliottower mgoulao realquantumcookie jjshoots standardgalactic spiglerg asadjeewa techthiyanes ethicalsecurity-agency kaleabtessera user402503 dearborn-open-ai joachimsalim fenggefeifei

shimmy's Issues

API break with gym 26.0

gym 26.0 new release removed the seed function, and instead added seed argument to reset.
https://github.com/openai/gym/releases
Now any code using sb3, shimmy, and gym>=26.0 raise an error since seed function does not exist, plus the seed functionality itself is not working properly.

Thanks!

Failing Environments

These environments are failing not because of the wrappers, but because the underlying environments are stupid.

DM_Control:

LQR (failing because of underlying environment instability)

OpenSpiel:

Side notes:

"nfg_game" isn't even registered under the python bindings.
"goofspiel" has fails for some ungodly reason.
same with "oshi zumo"

Bug in Shimmy v1.3.0 with DM-Control 1.0.15+: IndexError in 'torch.nn.modules.flatten'

I'm encountering a bug in Shimmy version 1.3.0 when using DM-Control 1.0.15 onwards (earlier vers had different errors). Here's the code I'm using:


import gym
from shimmy.registration import DM_CONTROL_SUITE_ENVS
from stable_baselines3 import PPO

env_ids = [f"dm_control/walker-stand-v0"]

env = gym.make(env_ids[0], render_mode="rgb_array")

env.reset()

model = PPO('MultiInputPolicy', env, verbose=1)
model.learn(total_timesteps=30000)

TIMESTEPS = 30000
for i in range(1, 30):
    model.learn(total_timesteps=TIMESTEPS, reset_num_timesteps=False, tb_log_name='PPO')
    model.save(f'./save/{TIMESTEPS*i}')

env.close()

The issue occurs when running the training agent on certain models, such as 'dm_control/walker-stand-v0', 'dm_control/humanoid-walk-v0', ...

The error message I'm getting is as follows:

  File "C:\Users\Admin\Desktop\dm control\gymDmc.py", line 32, in <module>
  
    **model.learn(total_timesteps=30000)**
  
  File "C:\Users\Admin\.conda\envs\shimmy\Lib\site-packages\stable_baselines3\ppo\ppo.py", line 315, in learn
    return super().learn(
           ^^^^^^^^^^^^^^
  File "C:\Users\Admin\.conda\envs\shimmy\Lib\site-packages\stable_baselines3\common\on_policy_algorithm.py", line 277, in learn
    continue_training = self.collect_rollouts(self.env, callback, self.rollout_buffer, n_rollout_steps=self.n_steps)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\.conda\envs\shimmy\Lib\site-packages\stable_baselines3\common\on_policy_algorithm.py", line 178, in collect_rollouts
    actions, values, log_probs = self.policy(obs_tensor)
                                 ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\.conda\envs\shimmy\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\.conda\envs\shimmy\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\.conda\envs\shimmy\Lib\site-packages\stable_baselines3\common\policies.py", line 643, in forward
    features = self.extract_features(obs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\.conda\envs\shimmy\Lib\site-packages\stable_baselines3\common\policies.py", line 670, in extract_features
    return super().extract_features(obs, self.features_extractor if features_extractor is None else features_extractor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\.conda\envs\shimmy\Lib\site-packages\stable_baselines3\common\policies.py", line 131, in extract_features
    return features_extractor(preprocessed_obs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\.conda\envs\shimmy\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\.conda\envs\shimmy\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\.conda\envs\shimmy\Lib\site-packages\stable_baselines3\common\torch_layers.py", line 277, in forward
    encoded_tensor_list.append(extractor(observations[key]))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\.conda\envs\shimmy\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\.conda\envs\shimmy\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\.conda\envs\shimmy\Lib\site-packages\torch\nn\modules\flatten.py", line 49, in forward
    return input.flatten(self.start_dim, self.end_dim)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Theres no error from other agents such as acrobot-swingup-v0, dog-walk-v0,..
How Should We Proceed to Fix This Issue? Thanks

Checklist for Maturity

Options are being ignored

The options keyword argument is being ignored in the Gym compatibility wrapper:

Shimmy/shimmy/openai_gym_compatibility.py

Line 231 in 6e82d3d

return self.gym_env.reset(), {}

Dependencies not fixed (pettingzoo issues)

Shimmy/setup.py

Line 49 in bd007d7

"meltingpot": ["pettingzoo>=1.22.3"],

@pseudo-rnd-thoughts we should probably fix these to specific versions at least for things like pettingzoo which could be updated and break things after release, id rather whatever shimmy release we have be internally compatible and work on its own, vs now it seems to be broken because it’s installing pettingzoo 1.23.0, when we’d be better off making a new shimmy release for that and fixing this 1.0.0 release at 1.22.3

Import issue: AttributeError: module 'pettingzoo' has no attribute 'AECEnv'

For some reason when running in my PettingZoo environment, I get this when I try to import gymnasium. It seems to be an underlying issue with shimmy, but I'm not sure why it doesn't happen when I run it in the shimmy repo itself or gymnasium repo.

My versions are as follows (running pip list in my virtual env):

Shimmy 0.2.1
gymnasium 0.28.1
pettingzoo 1.22.4

I tried pip install git+https://github.com/Farama-Foundation/Shimmy.git to get the most recent version and it didn't change things. I imagine some users will probably encounter this issue as well whatever the cause is.

import gymnasium
Traceback (most recent call last):
File "", line 1, in
File "/Users/elliottower/anaconda3/envs/pettingzoo/lib/python3.8/site-packages/gymnasium/init.py", line 12, in
from gymnasium.envs.registration import (
File "/Users/elliottower/anaconda3/envs/pettingzoo/lib/python3.8/site-packages/gymnasium/envs/init.py", line 382, in
load_plugin_envs()
File "/Users/elliottower/anaconda3/envs/pettingzoo/lib/python3.8/site-packages/gymnasium/envs/registration.py", line 600, in load_plugin_envs
fn = plugin.load()
File "/Users/elliottower/anaconda3/envs/pettingzoo/lib/python3.8/site-packages/importlib_metadata/init.py", line 208, in load
module = import_module(match.group('module'))
File "/Users/elliottower/anaconda3/envs/pettingzoo/lib/python3.8/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/Users/elliottower/anaconda3/envs/pettingzoo/lib/python3.8/site-packages/shimmy/init.py", line 43, in
from shimmy.openspiel_compatibility import OpenspielCompatibilityV0
File "/Users/elliottower/anaconda3/envs/pettingzoo/lib/python3.8/site-packages/shimmy/openspiel_compatibility.py", line 15, in
class OpenspielCompatibilityV0(pz.AECEnv):
AttributeError: module 'pettingzoo' has no attribute 'AECEnv'

Add release notes on website

Add support for Melting Pot scenarios

Currently, MeltingPotCompatibilityV0 works only when the input env is a substrate, rather than a scenario. This is because self._env.observation_spec()[0] has no attribute "WORLD.RGB". This can be rectified by replacing lines 90-92 of meltingpot_compability.py with

        if type(self._env) is meltingpot.utils.scenarios.scenario.Scenario:
            _env = self._env._substrate
        else:
            _env = self._env
        self.state_space = utils.dm_spec2gym_space(
            _env.observation_spec()[0]["WORLD.RGB"]
        )

and lines 129-131 with

        if type(self._env) is meltingpot.utils.scenarios.scenario.Scenario:
            _env = self._env._substrate
        else:
            _env = self._env
        observation_space = utils.remove_world_observations_from_space(
            utils.dm_spec2gym_space(_env.observation_spec()[0])  # type: ignore

However, the code still runs into a pickling error later down the line. What would be the best way to fix this and add the feature?

[Bug] Melting pot wrapper fails with newer versions of pettingzoo's tests.

Issue

Melting pot tests fail with newer versions of pettingzoo's tests.

Specifically this line,

assert set(infos.keys()) == (set(par_env.agents))

This is because info is not a dict per agent in the reset function as is expected by the test.

Shimmy/shimmy/meltingpot_compatibility.py

Line 179 in abf3122

return observations, {

PettingZoo Update breaks Shimmy

shimmy fails since pettingzoo was recently updated to remove ObsDict in version 1.24.0 from /pettingzoo/utils/env.py
ImportError: cannot import name 'ObsDict' from 'pettingzoo.utils.env'(/home/ajeewa/.local/lib/python3.11/site-packages/pettingzoo/utils/env.py)

Cannot use supersuit color_reduction_v0 with Shimmy

When i run the following code, the color_reduction_v0 wrapper fails with:

env = load_meltingpot("clean_up")
env = MeltingPotCompatibilityV0(env, render_mode="human")
env = color_reduction_v0(env, 'full') # grayscale

AttributeError: 'Dict' object has no attribute 'low'
Shimmy is a great initiative :D

Render modes issues in tests

Currently, skip_render_check is enabled for all tests in tests/test_gym.py because GymV26Compatibility doesn't inherit the sub_class' render_modes.

skip_render_check is also enabled for the wrapper tests in tests/test_dm_control.py because of this same issue I think. Although this should be fixed in the next gymnasium release.

Reasoning for rendering FPS for dm_control environments

Hello Shimmy Contributors,

I just wanted to ask about the reason behind the metadata['render_fps'] setting for dm_control environments.
In dm_control_compatibility.py#81, it seems like the render_fps is set to the environment's control_timestep converted to millisecond, but I think it is more natural to have it set to int(round((1/control_timestep))? I'm more than happy to submit a PR for this.

Improvements for dm_control Compatibility Wrapper

Control Frequency

Currently, the control frequency is exposed for the Mujoco environments of gymnasium like this

@property
def dt(self):
    return self.model.opt.timestep * self.frame_skip

With the dm_control compatibility wrapper of shimmy we can also access this information, however this is only possible via

import gymnasium as gym 
env = gym.make('dm_control/cartpole-balance-v0')
env.control_timestep()

For consistency and ease of use, I would propose to expose this information the same way, e.g. by adding a property to the compatibility class

@property
def dt(self):
    return self._env.control_timestep()

TimeLimit

Additionally, for ease of use it would also be beneficial to have the defined time limits as part of the environment spec, currently this value is just None.
This information could be retrieved and added to the environment spec in the __init__ e.g.

if self.env_type is env_type.COMPOSER:
    self.spec.max_episode_steps = ceil(self._env._time_limit / self._env.control_timestep())
elif self.env_type is env_type.RL_CONTROL:
    self.spec.max_episode_steps = int(self._env._step_limit)

[Feature] External APIs to support

This issue aims to list all of the API that are not Gymnasium or PettingZoo

For APIs missing from this list that you wish us to support, please comment and we will try to update the list

MeltingPotCompatabilityV0 & supersuit concat_vec_envs

Hi,

Do we know if the MeltingPot parallel envs are compatible with the parallel vector envs in supersuit?

Currently using the following:

    env = load_meltingpot(args.env_id)
    env = DownSamplingSubstrateWrapper(env, 8)
    env = MeltingPotCompatibilityV0(env, render_mode="rgb_array")
    env = ss.pettingzoo_env_to_vec_env_v1(env)
    env = ss.concat_vec_envs_v1(env, args.num_envs, num_cpus=1, base_class='gymnasium')

throws up the error:

TypeError: cannot pickle 'dmlab2d.dmlab2d_pybind.Lab2d' object

however I might be completely missing something, please let me know!

Add Nvidia Omniverse Isaac Envs

Isaac gym has support for OpenAI Gym v0.24.1, it would be nice to update to Gymnasium
https://github.com/NVIDIA-Omniverse/IsaacGymEnvs

This PR includes support, to investigate why it is needed
https://github.com/Toni-SM/skrl/tree/main/skrl/envs/torch

meaning of gym v21

Hi,

I just wonder that the documentation of using v21 and v26. Here v21 means environment from v21 to v25 or only strictly limited to v21?

thanks your great work.

Possibility to render multiple cameras at the same time in dm_control environments

Hello, I think it is possible to render multiple cameras at the same time in dm_control by composing different calls to the physics.render() method. Can this be an option in DmControlCompatibilityV0? I can try to make a pull request

Not import all compatibility wrappers in `init.py`

Currently in shimmy.__init__.py, we import all compatibility wrappers however this can be time consuming to import all of these modules. I propose adding something like this Farama-Foundation/Gymnasium#392

Broken dm_control.viewer

When I try using shimmy with locomotion (uses composer.Environment) or a custom composer.Environment env, the dm_contorl.viewer starts but is either unresponsive (with default locomotion envs) or has no animation (for my custom env)

But the env is working, as you can see observations and rewards changing. No issues with suite envs (control.Environment).

from shimmy.dm_control_compatibility import DmControlCompatibilityV0
from dm_control.locomotion.walkers.cmu_humanoid import CMUHumanoidPositionControlled
from dm_control.locomotion.arenas.floors import Floor
from dm_control.locomotion import tasks
from dm_control import composer

task = tasks.corridors.RunThroughCorridor(CMUHumanoidPositionControlled(), Floor())
env = DmControlCompatibilityV0(composer.Environment(task), render_mode="human")
observation, info = env.reset(seed=42)
for _ in range(1000):
   action = env.action_space.sample()  # this is where you would insert your policy
   observation, reward, terminated, truncated, info = env.step(action)
   print(reward)

   if terminated or truncated:
      observation, info = env.reset()
env.close()

OS: Linux-5.19.0-43-generic-x86_64-with-glibc2.35 # 44~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon May 22 13:39:36 UTC 2
Python: 3.10.6
PyTorch: 2.0.1+cu117
GPU Enabled: True
Numpy: 1.24.3
Cloudpickle: 2.2.1
Gymnasium: 0.28.1
OpenAI Gym: 0.26.2
Mujoco: 2.3.5
dm-control: 1.0.12
Shimmy: 0.2.1

Add support for dm ai-safety-gridworlds

Add support for https://github.com/deepmind/ai-safety-gridworlds
We have a number of DM environments already so this should be too bad ... hopefully.

[Issue] Gym Compatibility environment calling functions on the base environment

Based on Farama-Foundation/Gymnasium#162 there are issues that we need to access parameters/functions on the base environment

Add __getattr__ should work for accessing parameters, however for functions I am not certain if this will work or if we need another functions
Add testing to check this

Render issues

import gymnasium as gym

def make_env(env_id, seed, idx, capture_video, run_name, gamma):
    def thunk():
        if capture_video:
            env = gym.make(env_id, render_mode="rgb_array")
        else:
            env = gym.make(env_id)
        env = gym.wrappers.RecordEpisodeStatistics(env)
        if capture_video:
            if idx == 0:
                env = gym.wrappers.RecordVideo(env, f"videos/{run_name}")
        env.action_space.seed(seed)
        env.observation_space.seed(seed)
        return env

    return thunk

envs = gym.vector.SyncVectorEnv([make_env("dm_control/walker-run-v0", 0, 0, True, "videos", 0.99)] )
envs.reset()
for i in range(1000):
    envs.step(envs.action_space.sample())

x observation space has an unconventional shape (neither an image, nor a 1D vector). We recommend flattening the observation to have only a 1D vector or use a custom policy to properly process the data. Actual observation shape: ()
  logger.warn(
Moviepy - Building video /home/costa/Documents/go/src/github.com/cleanrl/cleanrl/videos/videos/rl-video-episode-0.mp4.
Moviepy - Writing video /home/costa/Documents/go/src/github.com/cleanrl/cleanrl/videos/videos/rl-video-episode-0.mp4

Moviepy - Done !                                                                                                                                             
Moviepy - video ready /home/costa/Documents/go/src/github.com/cleanrl/cleanrl/videos/videos/rl-video-episode-0.mp4
Traceback (most recent call last):
  File "/home/costa/Documents/go/src/github.com/cleanrl/cleanrl/minimal_bug.py", line 22, in <module>
    envs.step(envs.action_space.sample())
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gymnasium/vector/vector_env.py", line 197, in step
    return self.step_wait()
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gymnasium/vector/sync_vector_env.py", line 154, in step_wait
    observation, info = env.reset()
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gymnasium/wrappers/record_video.py", line 94, in reset
    observations = super().reset(**kwargs)
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gymnasium/core.py", line 402, in reset
    return self.env.reset(**kwargs)
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gymnasium/wrappers/record_episode_statistics.py", line 69, in reset
    obs, info = super().reset(**kwargs)
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gymnasium/core.py", line 402, in reset
    return self.env.reset(**kwargs)
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gymnasium/wrappers/order_enforcing.py", line 42, in reset
    return self.env.reset(**kwargs)
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gymnasium/wrappers/env_checker.py", line 47, in reset
    return self.env.reset(**kwargs)
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/shimmy/dm_control_compatibility.py", line 83, in reset
    timestep = self._env.reset()
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/dm_control/rl/control.py", line 81, in reset
    with self._physics.reset_context():
  File "/home/costa/.pyenv/versions/3.9.5/lib/python3.9/contextlib.py", line 117, in __enter__
    return next(self.gen)
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/dm_control/rl/control.py", line 242, in reset_context
    self.reset()
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/dm_control/mujoco/engine.py", line 289, in reset
    mujoco.mj_resetData(self.model.ptr, self.data.ptr)
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/dm_control/mujoco/engine.py", line 531, in model
    return self._data.model
AttributeError: 'Physics' object has no attribute '_data'

Circular dependency between gymnasium and shimmy

Hey there, I saw shimmy has a dep of gymnasium and gymnasium in the mean time has a dep of shimmy. That's OK for most of the cases, but it blocks envpool's recent build and test because bazel cannot accept this kind of circular dep. Is it possible to fix this issue in either gymnasium or shimmy? Thanks!

Update: right now I have to specify this to unblock myself:

gymnasium>=0.26,!=0.27.0

[Bug] incompatible with latest version of Meltingpot

Issue

If you install the latest meltingpot version (pip install dm-meltingpot or just clone and install mp) and run the following code:

from meltingpot import substrate

def main():
    from shimmy import MeltingPotCompatibilityV0
    from shimmy.utils.meltingpot import load_meltingpot

    env = load_meltingpot("clean_up")
    env = MeltingPotCompatibilityV0(env, render_mode="rgb_array")

    observations = env.reset()
    while env.agents:
        actions = {agent: env.action_space(agent).sample() for agent in env.agents}
        observations, rewards, terminations, truncations, infos = env.step(actions)
    env.close()


if __name__ == "__main__":
  main()

You get this error - AttributeError: module 'meltingpot' has no attribute 'python' from the line here

 player_roles = meltingpot.python.substrate.get_config(

This is because MP has changed how things are aliased (no longer using meltingpot.python).

Check when importing gym wrappers

thu-ml/tianshou#789

farama-foundation / shimmy Goto Github PK

shimmy's People

Contributors

Stargazers

Watchers

Forkers

shimmy's Issues

Checklist for Maturity

Issue

Control Frequency

TimeLimit

Issue

Recommend Projects

Recommend Topics

Recommend Org