Markov decision processes studies sequential decision making for stochastic processes. Here the case of Markov chain where policy decisions are determined in order to minimize the long term costs. It naturally extends the stochastic dynamic programming to the long term stationary case.

For further studies in this area you may consider EL2805 Reinforcement Learning and SF2852 Optimal Control Theory..

Here are pdf-files for the Markov Decision Process prototype example and the various solutions to this problem that will be presented at the last two lectures:

LectureMDP2025.pdf (Updated!)

With an alternative derivation of the VDE for the problem without discounting.

Videos from Covid times

Reading instructions

Lecture 13: Markov Decision Processes, without discount

9th edition : Sections 19.4.

10th, 11th edition: supplement 1 (Links to an external site.)

Lecture 14: Markov Decision Processes, with discount

9th edition : Sections 19.5.

10th, 11th edition: supplement 2