Comments (5)
Hey reporting back here. After using built wheels that fixes the clipped reward bug here: https://github.com/sail-sg/envpool/actions/runs/1690252100, I was able to reproduce PPO's old performance in Breakout. (See the blue line)
Thanks @Trinkle23897 for the helpful fix
from envpool.
Got some feedback from @JesseFarebro regarding envpool's Breakout-v5
vs ALE/Breakout-v5
:
- They
envpool
aren't doing terminal signal on loss of life which in Breakout is the single most important setting. (we don't recommend it be used)- They are doing NOOPs without sticky actions and you're doing both (no one should be using NOOPs anymore).
- They don't do reward clipping by default? Check the magnitude of the rewards if you aren't clipping yourself (this is fine)
As shown in the script, I was already doing
envs = envpool.make(
args.gym_id,
env_type="gym",
num_envs=args.num_envs,
episodic_life=True,
reward_clip=True,
)
Maybe the reason is "They are doing NOOPs without sticky actions and you're doing both (no one should be using NOOPs anymore)."?
from envpool.
https://envpool.readthedocs.io/en/latest/api/atari.html
(no one should be using NOOPs anymore)."?
set noop_max=0
from envpool.
Hmm but the BreakoutNoFrameskip-v4
result clearly used the noop_max
.
def make_env(gym_id, seed, idx, capture_video, run_name):
def thunk():
env = gym.make(gym_id)
env = gym.wrappers.RecordEpisodeStatistics(env)
if capture_video:
if idx == 0:
env = gym.wrappers.RecordVideo(env, f"videos/{run_name}")
env = NoopResetEnv(env, noop_max=30)
env = MaxAndSkipEnv(env, skip=4)
env = EpisodicLifeEnv(env)
if "FIRE" in env.unwrapped.get_action_meanings():
env = FireResetEnv(env)
env = ClipRewardEnv(env)
env = gym.wrappers.ResizeObservation(env, (84, 84))
env = gym.wrappers.GrayScaleObservation(env)
env = gym.wrappers.FrameStack(env, 4)
env.seed(seed)
env.action_space.seed(seed)
env.observation_space.seed(seed)
return env
return thunk
from envpool.
Per conversation with @Trinkle23897, we think the problem is reward counting. In my implementation, I counted the rewards from all five lives, but since I was using reward_clip=True
in envpool.make
, I was counting the clipped rewards from 5 episodes. Maybe the best solution is to include a game_score
key in the info
variable.
class RecordEpisodeStatistics(gym.Wrapper):
def __init__(self, env, deque_size=100):
super(RecordEpisodeStatistics, self).__init__(env)
self.num_envs = getattr(env, "num_envs", 1)
self.episode_returns = None
self.episode_lengths = None
# get if the env has lives
self.has_lives = False
env.reset()
info = env.step(np.zeros(self.num_envs, dtype=int))[-1]
if info["lives"].sum() > 0:
self.has_lives = True
print("env has lives")
def reset(self, **kwargs):
observations = super(RecordEpisodeStatistics, self).reset(**kwargs)
self.episode_returns = np.zeros(self.num_envs, dtype=np.float32)
self.episode_lengths = np.zeros(self.num_envs, dtype=np.int32)
self.lives = np.zeros(self.num_envs, dtype=np.int32)
self.returned_episode_returns = np.zeros(self.num_envs, dtype=np.float32)
self.returned_episode_lengths = np.zeros(self.num_envs, dtype=np.int32)
return observations
def step(self, action):
observations, rewards, dones, infos = super(RecordEpisodeStatistics, self).step(
action
)
self.episode_returns += rewards
self.episode_lengths += 1
self.returned_episode_returns[:] = self.episode_returns
self.returned_episode_lengths[:] = self.episode_lengths
all_lives_exhausted = infos["lives"] == 0
if self.has_lives:
self.episode_returns *= (1 - all_lives_exhausted)
self.episode_lengths *= (1 - all_lives_exhausted)
else:
self.episode_returns *= (1 - dones)
self.episode_lengths *= (1 - dones)
infos["r"] = self.returned_episode_returns
infos["l"] = self.returned_episode_lengths
return (
observations,
rewards,
dones,
infos,
)
from envpool.
Related Issues (20)
- [Question] Does EnvPool support Procgen's Exploration mode
- [Feature Request] 请问什么时候才能在windows上使用呀
- Does envpool support arm architecture? HOT 2
- [BUG] AssertionError: MiniGrid-Empty-5x5-v0 is not supported HOT 2
- Set values for action in C++ HOT 2
- [BUG] XLA interface does not work with jax>=0.4.16
- [BUG] XLA Segmentation Fault HOT 8
- [BUG] No Envs after installing from sources HOT 19
- metaclass conflict HOT 1
- [BUG] Unable to build from source HOT 6
- [Feature Request] Add Atari difficulty level and a game mode option. HOT 3
- [BUG] Build failes due to Bazel 7.0.0 breaking change
- output 'envpool/mujoco/assets_gym' of //envpool/mujoco:gen_mujoco_gym_xml is a directory; dependency checking of directories is unsound
- [Feature Request] A simple (and effective?) way to support cherry-picked env reset in `xla` mode HOT 1
- [Feature Request] Get RAM State from Atari ALE HOT 5
- [BUG] Error running mujoco-gym tasks with the parameter xml_file HOT 1
- [BUG] incorrect parsing of actions in multiagent environment
- [Feature Request]Set random seed to each env HOT 1
- [BUG] Episode return is not recorded correctly in cleanRL's example
- Opinions on running envpool on a dedicated simulator server with e.g. REST API HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from envpool.