Markov Decision Process
- Definition
- A Markov Decision Process (MDP) formalizes decision-making under uncertainty: states, actions, transition probabilities, and rewards. An agent chooses actions to maximize expected cumulative reward over time, balancing immediate gains with long-term benefits. This differs from passive Markov chains because an MDP includes deliberate choices (actions) and a reward signal — it's the foundation of many reinforcement learning methods. Think of it as a game engine where you design policies (action rules) to get the best score given stochastic dynamics.
How does it work?
Markov Decision Process algorithms typically represent uncertainty explicitly and use probability calculations to combine data and model assumptions. Implementations focus on numerical stability, how to represent distributions (parametric vs sampled), and efficient updates when new data arrive.
Examples
- Robotics navigation — Model robot movement and rewards to compute optimal policies for reaching goals while avoiding obstacles.
- Inventory management — Use states for stock levels and actions for reorder decisions to minimise costs under uncertainty.
- Automated trading backtests — Model discrete states and actions to evaluate policy performance under simulated market dynamics.
Problems
- Defining a reward function that doesn't produce unintended behavior (reward hacking)
- Curse of dimensionality in large state/action spaces
- Uncertainty or inaccuracy in the transition model
- Balancing discount factor choice against convergence and myopia
- Difficulty validating the policy before deployment in a real environment