Q-Learning
- Definition
- Q-Learning is a model-free reinforcement learning algorithm that learns the value (Q) of taking actions in states by bootstrapping from observed rewards and future value estimates. Think of updating a table that scores how good each action is in each situation through trial and error. Q-Learning differs from policy gradient methods by learning value estimates directly and from model-based approaches by not requiring a model of environment dynamics.
How does it work?
Q-Learning operates by interacting with an environment: observe state, choose action, receive reward, and update a policy or value estimator. Practical implementations must balance exploration and exploitation and handle credit assignment across time via bootstrapping or Monte Carlo returns.
Examples
- Grid-world navigation — Tabular Q-learning to teach agents to reach goals in discrete environments.
- Game AI (classic) — Train agents to play simple arcade games using Q-value updates and epsilon-greedy exploration.
- Ad placement bandits — Use Q-learning variants to learn action values for contextual decision settings.
Problems
- Q-table becomes intractable in large or continuous state spaces
- Slow convergence, requiring many episodes of exploration
- Balancing exploration vs. exploitation (epsilon tuning)
- Overestimation bias in Q-value updates
- Sensitive to reward shaping and hyperparameters