Markov Chain
- Definition
- A Markov chain models a process that moves through states step by step where each step depends only on the current state, not the full past. Practically, it's like a simple rule that says “given where I am now, here are the chances of moving to each next place.” This differs from more complex sequence models (like hidden Markov models) because a plain Markov chain assumes you can observe the states directly and don't need to infer hidden information. Use it when the "memory" of the system is limited to the present.
How does it work?
Markov Chain algorithms typically represent uncertainty explicitly and use probability calculations to combine data and model assumptions. Implementations focus on numerical stability, how to represent distributions (parametric vs sampled), and efficient updates when new data arrive.
Examples
- Practical inference — Estimate parameters or states from noisy data in applied systems.
- Sensor fusion — Combine multiple noisy measurements into a single estimate.
- A/B analysis — Update beliefs about variants using observed outcomes.
Problems
- Violated Markov (memoryless) assumption when real dependencies span multiple past states
- State space explosion making the transition matrix intractable
- Estimating transition probabilities reliably from sparse data
- Determining an appropriate order for the chain