Introduction to the POMDPs.jl framework and its ecosystem.

The course covers how to build and solve decision making problems in uncertain environments using the POMDPs.jl ecosystem of Julia packages. Topics covered include sequential decision making frameworks—namely, Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs)—running simulations, online and offline solution methods (value iteration, Q-learning, SARSA, and Monte Carlo tree search), reinforcement learning, deep reinforcement learning (including proximal policy optimization (PPO), deep Q-networks (DQN), and actor-critic methods), imitation learning through behavior cloning of expert demonstrations, state estimation through particle filtering, belief updating, alpha vectors, approximate methods (including grid interpolation for local approximation value iteration), and black-box stress testing to validate autonomous systems. The course is intended for a wide audience—no prior MDP/POMDP knowledge is expected.

1. Install Julia (we used v1.6.2, other versions should work)
2. Install Pluto.jl
3. Clone this repo:

   git clone https://github.com/JuliaAcademy/Decision-Making-Under-Uncertainty

4. From the Julia REPL (julia), run Pluto (a web browser window will pop-up):

   julia> using Pluto
   julia> Pluto.run()

   Or you can simply run the following in a terminal:

   julia -E "using Pluto; Pluto.run()"

5. From Pluto, open one of the .jl notebook files located in the Decision-Making-Under-Uncertainty/notebooks/ directory—enjoy!

The lectures can be found on Julia Academy and YouTube. They are broken down as follows.

Brief introduction to the content of this course.

Introduction to MDPs using the Grid World problem.

Introduction to POMDPs using the Crying Baby problem.

Using beliefs to estimate the state of an agent through particle filtering.

Approximating a continuous space using grid interpolation and value function approximation.

Introduction to deep reinforcement learning applied to the pendulum swing-up MDP.

Introduction to imitation learning using behavior cloning of expert demonstrations.

Stress testing a black-box system using adaptive stress testing.

Created and taught by Robert Moss.