Policy Gradient Method
- Definition
- Policy Gradient methods directly optimize the agent's action-selection strategy by estimating gradients of expected reward with respect to policy parameters and taking gradient steps. Picture nudging the policy toward actions that led to higher rewards. These methods differ from value-based ones (like Q-learning) as they optimize the policy directly and can naturally handle continuous action spaces.
How does it work?
Policy Gradient Method operates by interacting with an environment: observe state, choose action, receive reward, and update a policy or value estimator. Practical implementations must balance exploration and exploitation and handle credit assignment across time via bootstrapping or Monte Carlo returns.
Examples
- Continuous control (robotics) — Optimise stochastic policies directly for torque/actuator control using REINFORCE or PPO.
- Dialogue policy learning — Train end-to-end dialogue managers that select responses to maximise user satisfaction.
- Portfolio optimisation — Learn stochastic allocation policies to trade off risk and return in simulation.
Problems
- High variance in gradient estimates, slowing convergence
- Sample inefficiency requiring many environment interactions
- Sensitive to learning rate — small changes can destabilize training
- Difficulty designing reward signals that avoid unintended policies
- Can converge to poor local optima