This is a repository which contains simple implementations for ME5406 Deep learning for robotics

Consider a RL based robot is in a grid frozen lake environment, where the goal of the robot is to pick up the target while avoid falling into the ice holes

1. First visit monte carlo
2. SARSA
3. Q-learning

• numpy
• matplotlib
• tkinter
• PIL

Need to define the RL agent type ('mc'/'sarsa'/'ql'), the grid world size (4/10), the number of training epochs.

For example, if you want to train q-learning with 4x4 grid world with 10000 epochs, then you could use

python train.py --agent 'lq' --grid_size 4 --num_epoch 10000

Need to define the test type, which can be described as below

• Job 0: 4x4 frozen lake environment training, correctness test, and comparison test among algorithms
• Job 1, 10X10 frozen lake environment training, correctness test, and comparison test among algorithms
• Job 2, comparison test for different learning rate value settings
• Job 3, comparison test for different gamma value settings
• Job 4, comparison test for different epsilon value settings

If you want to test with Job 0, you could use

python train.py --job 0 --num_epoch 10000

• First visit Monte Carlo

  • 4x4 grid world
  • 10x10 grid world

• SARSA

  • 4x4 grid world
  • 10x10 grid world

• Q-learning

  • 4x4 grid world
  • 10x10 grid world

• Final policy

  • 4x4 grid world From left to right: Monte Carlo, SARSA, Q-learning

  • 10x10 grid world From left to right: Monte Carlo, SARSA, Q-learning

• Algorithm comparisons

  • 4x4 grid world
  • 10x10 grid world

• Learning rate Red/0.01, Green/0.001, Blue/0.0001

• Gamma Red/0.8, Green/0.9, Blue/0.99

• Epsilon Red/0.7, Green/0.8, Blue/0.9

Note: I didn't put all the plots here, but you can find all of them in the ./Results file

Morvan Zhou

Don't forget to give me a star if you like this! 😊