This is a tool for graphically illustrating the (possibly time-dependent) Markov chain underlying a Markov Decision Process (MDP).

First define the transition probabilities of the MDP in mdp.py along with the number of actions and the state labels. A variant of the standard machine replacement problem is provided as an example in the file.

To run MDP-Explore, type pythonw main.py.

The states will be plotted along with the transitions. If Show probabilities is enabled, then the transition probabilities will be added as labels to the transitions. The thicknesses of the arrows is proportional to the transition probabilities. You can choose which action and time that should be used to evaluate the transition matrix.

The current plot is saved and can be found in the img/ folder.

MDP-Explore requires graphviz and wxPython to run.

On OSX, these can be installed by running brew install graphviz, pip install graphviz and brew install wxpython. If you run into problems with linking libpng, try running sudo chown -R $USER:admin /usr/local/share/man/man5 (see this).