Markov processes are probably the most used models for describing systems with a certain degree of randomness. Some students may have taken the course SF1904 Markov Processes, it is very helpful if you have, but it is not a requirement to take this course.

Since all students have not studied Markov theory, and some need a quick reminder, we will spend a little time on introducing basic concepts and focusing on some of the mathematical theory needed later in the course.

Here is a link to the slides used in the lectures on Markov Chains:

LectureMC2025.pdf

Here is a link to the matlab script I used to demonstrate what happens with the state probabilities in steady state. And here is the second script I used for continuous time processes

Videos from Covid times

Lecture 1
- Markov Chains - Markov property , Stationary,
- Matlab exampleLinks to an external site.
Lecture 2
- Markov Chains - State characterization (Links to an external site.), Stationary state probabilities (Links to an external site.), Examples (Links to an external site.), Extra (Links to an external site.)
- Markov Processes - State characterization , Stationary state probabilities (Links to an external site.), Examples
Lecture 3
- Dwell times, Exponential distribution, Jump probabilities
- Poisson process, Hitchhiker paradox

Reading instructions

Lecture 1: Course introduction. Markov chains
9th edition: Sections 1.1-1.3, 16.1-3.
10th edition: Sections 1.1-1.3, 29.1-3 (Links to an external site.),

11th edition: Sections 1.1-1.3, 28.1-3
Lecture 2: Markov processes
9th edition: Sections 16.4-16.5, 16.8.
10th edition: Sections 29.4-5, (29.6-7), 29.8

11th edition: Sections 28.4-5, (28.6-7), 28.8

Lecture 3: Exponential distribution, Poisson process, Sections 17.4.

Additional literature: Markov kompendium (På svenska från MatStat), Essentials of Stochastic Processes by Durrett.

The first home assignment is an application of Markov theory to reliability theory.