Taking actions inside emulator about gym-super-mario HOT 26 CLOSED

The action parameter is an array in the format [up, left, down, right, A-button, B-button].

e.g.

env = gym.make('ppaquette/SuperMarioBros-1-1-v0')

# This loads the game and skips the intro, and then freezes until it receives a step
env.reset()

obs, rew, is_done, info = env.step([0,0,0,1,0,0])         # Move right
obs, rew, is_done, info = env.step([0,0,0,1,1,0])         # Move right while jumping
obs, rew, is_done, info = env.step([0,0,0,1,0,1])         # Move right while running

I haven't had the chance to test the env under MacOS, but it works well under Ubuntu.

from gym-super-mario.

Were you able to get it working through a script?

The subprocess.popen seems to be blocking inputs on the terminal.

I might be able to script a workaround if you are not able to send actions through a python script.

from gym-super-mario.

Hey Philip, sorry for the delay. I can't enter any inputs into the terminal, probably because of the issue you spotted. Have you been able to sort a workaround?

from gym-super-mario.

Sorry for the delay.

Run the following (with both gym-pull and ppaquette-gym-super-mario uninstalled) and let me know the output.

test.py is available here

pip install gym-pull
python -c "import gym; import gym_pull; gym_pull.pull('github.com/ppaquette/gym-super-mario@dev')"
python test.py

from gym-super-mario.

Hi Philip, sorry again for the delay. Here's the output that I got from running test.py. After this point, the emulator started beachballing and nothing happened, so I force quit after a few minutes.

Gabes-MacBook-Pro-2:Desktop ggrand$ python test.py
-- START creating environment
[2016-11-05 20:15:26,037] Making new env: ppaquette/SuperMarioBros-1-3-v0
-- DONE creating environment
-- START resetting environment
[2016-11-05 20:15:26,044] -- START _reset()
[2016-11-05 20:15:26,044] -- START resetting variables
[2016-11-05 20:15:26,044] -- DONE resetting variables
[2016-11-05 20:15:26,044] -- START launching fceux
[2016-11-05 20:15:26,044] -- START creating pipes
[2016-11-05 20:15:26,045] -- DONE creating pipes
[2016-11-05 20:15:26,045] -- START creating lua file
[2016-11-05 20:15:26,046] -- DONE creating lua file
[2016-11-05 20:15:26,046] -- START popen fceux
[2016-11-05 20:15:26,050] -- DONE popen fceux
[2016-11-05 20:15:26,050] -- START opening OUT pipe

from gym-super-mario.

Was the 1-3 map loaded or was it the 1-1 demo?

The problem is that the output pipe (to send commands from python to fceux) is created, but not opened by fceux, so Python hangs until the fifo pipe is opened by a reader (i.e. never).

The python code uses tempfile.gettempdir(), while the lua code assumes this means /tmp. Your temp directory might be located in /var/tmp or /usr/tmp.

I should have a patch by end of day.

from gym-super-mario.

The 1-3 map is the one that loads. On my mac, the tmp folder is located in /usr/tmp, and there is no folder called /var/tmp.

Thanks so much for your quick response. Please let me know if there's anything else you need me to test.

from gym-super-mario.

I've published a tentative patch on the dev branch. Uninstall both gym-pull and ppaquette-gym-super-mario uninstalled, and run the same commands as last time.

My output is: fceux.txt

from gym-super-mario.

Awesome, now the actions seem to be working! Unfortunately, on several of the runs, the emulator froze, so I had to force quit. When I did, I got the error message:

[2016-11-08 17:40:36,827] Closing episode (appears to be stuck). See documentation for how to handle this issue.
{'ignore': True}

This issue seems to occur only after Mario has already died once. If I quit the terminal and re-run the script, the first life runs smoothly, and then the emulator freezes shortly into the next life.

EDIT: Freezing sometimes occurs on the first life, as well.

from gym-super-mario.

Hi Philip, do you have any advice for how to handle the freezing? I tried using a multiprocessing lock, as recommended in the readme, but I'm not sure if I set it up right.

from gym-super-mario.

I've pushed a tentative fix on the 'dev' branch, but I can't really reproduce the bug.

Let me know if the fix works. Otherwise, I'll need the code you are running that produces the bug. Running the env in a loop with a reset() does not make the game freeze on my computer (but the game usually takes 2-3 seconds to fully start)

from gym-super-mario.

I just updated the env with the code on the dev branch, but the update seems to have made it worse :/ Previously, it got stuck about 1/4 iterations, and now it gets stuck every time.

Here's the repo that I've been working on...
https://github.com/kevinloughlin/Super-Mario-RL

from gym-super-mario.

Can't reproduce.

Try to use the code in the master branch (i.e. the code that should be crashing 1/4), but run gym.make and env.close() inside your loop, to fully create and destroy the env on every iteration.

from gym-super-mario.

I've switched back to the master branch, and I'm creating and destroying the env on every iteration, as you suggested. It's back to freezing about 1/4 of the time, which should be serviceable.

You might be interested to know that on one occasion only, when the emulator froze, I got the following traceback:

Exception in thread Thread-8:
Traceback (most recent call last):
  File "//anaconda/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "//anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "//anaconda/lib/python2.7/site-packages/ppaquette_gym_super_mario/nes_env.py", line 163, in _listen_to_incoming_pipe
    self._process_pipe_message(buffer[:-1])
  File "//anaconda/lib/python2.7/site-packages/ppaquette_gym_super_mario/super_mario_bros.py", line 188, in _process_pipe_message
    self._process_screen_message(frame_number, data)
  File "//anaconda/lib/python2.7/site-packages/ppaquette_gym_super_mario/super_mario_bros.py", line 108, in _process_screen_message
    x = int(part[0:2], 16)
ValueError: invalid literal for int() with base 16: 'n_'

from gym-super-mario.

This is likely the bug I was looking for.
Pushed a patch on the master branch, let me know if this fixes the issue.

from gym-super-mario.

Agh, so close, but not quite. I used gym-pull to get the master branch, and ran my Q learner on the first three worlds, for 10 iterations each. The game froze 4 / 30 iterations, which is not bad at all, but it also means that the issue is not totally resolved. (Incidentally, is there any way to speed up the emulator during training?)

Please let me know if there's anything I can do to get more insight on the cause of the freezing. Otherwise, I'm going to post some other questions I have in a different thread, but I'll leave this one open in case you come up with anything else on your end.

from gym-super-mario.

Added another try/catch block to prevent the same issue. Let me know if this reduces further the number of freezes.

from gym-super-mario.

As for the emulator being slow, I'm sending the value of all the modified pixels through the pipe, which is definitely the bottleneck. Any idea on how to better implement it?

from gym-super-mario.

Still getting some freezing, especially when there are other processes running on my machine (e.g., when I'm trying to record the game with a screen recording). I really appreciate your continued efforts to iron this out though.

As for the speed of the emulator, I want to make sure that I understand where you're coming from. I noticed that at the beginning of the game, it says the emulator is running at 3200% speed. However, to me, the game looks to be running at normal speed. I'm not sure whether that's because the emulator is slow, or because it isn't actually running at 3200% speed. I did notice that the game audio is basically reduced to white noise, presumably because the emulator is sped up (It sounds normal when playing with human inputs using the env wrapper). I also notice that there's a slowdown as soon as the actions start getting piped in. So again: either the emulator is running at 3200% speed but the bottleneck is making it look like it's running at normal speed, or the emulator isn't running at 3200% speed in the first place. What do you think?

from gym-super-mario.

Not quite.

1. Usually fceux (in normal mode) runs at ~60 fps. fceux adds a delay between frames to get this fps. With 3200%, the delay is removed, so fceux is able to run (in theory) up to ~200 fps.

2. gym envs normally run at 30-35 fps, so I'm processing a frame, and repeating the action for another frame to get this fps (60 fps / 2 frames per step = 30 effective fps).

3. On every processed frame, I send the pixel value for all changed pixels (i.e. if Mario moves to the left or right, most pixels change value) through the fifo pipe. They are concatenated together for every pixel row on the screen.

4. There are 256x224 pixels on the fceux screen, so I need to send up to 57,344 pixel values (RGB) 30 times per second. (I'm only sending the pixel value if they have changed since the last frame). The pipe is not able to process that many messages per second, so it lags.

The lag is worse if the background is complex (e.g. a castle), or if other processes are running on the computer. So the game is in theory running at 200 fps, but the pipe is slowing it down to a (visual) 15-30 fps.

Any idea how I could better implement this?

e.g. I could only send the tiles value (and not the screen pixels value) for Tiles world, but that would make recording a video of gameplay not possible.

from gym-super-mario.

I'm with you, up to the part where you said that sending only the tiles values through the pipe (that was what I was going to suggest) would make it not possible to see the gameplay. Isn't the purpose of sending the pixels through the pipe to give the user access to the pixel RGBs in order to do learning / action selection? Why does the emulator care that the user isn't getting the pixels?

from gym-super-mario.

1. Usually gym envs display the screen through env.render()
2. fceux actually displays the RGB pixel values through the fceux process, so there is no need to call render()
3. for envs (i.e. non-Tiles) that required RGB pixels, we obviously need to send them through the pipe for learning
4. for Tiles world, we need to send the 16x13 tiles through the pipe for learning, but also send the 256x224 RGB value so the gym recorder can create a video recording to upload to the scoreboard.

My suggestion was that for the video recording portion of the Tiles env, we could use the 16x13 Tiles rather than the 256x224 screen, so we wouldn't need to send all the pixels on the pipe.

from gym-super-mario.

That sounds great to me. I was going to use QuickTime player to record the video anyway, so there's no need on my end to have the gym recorder have access to all the pixels. At least when we upload to the scoreboard, the gym recorder will be able to use the tiles. Hopefully not having to send all the pixels through the pipe the will greatly improve the emulator speed, at least for the tiles version.

from gym-super-mario.

Published a patch. Definitely much faster for tiles env. Let me know what you think of it.

from gym-super-mario.

Wow! So much smoother! That also seems to have (fingers crossed) fixed the freezing issues as well. You deserve a medal.

from gym-super-mario.

Hello! I have a pretty simple question. How do you access the 16x13 Tiles to do learning?

from gym-super-mario.