Gradient Descent
How does it work?
Gradient-based methods compute the gradient of an objective with respect to parameters and take steps in the negative gradient direction. Key practical aspects are step-size (learning rate), batch vs full gradients, and techniques like momentum or adaptive optimizers (Adam) to improve convergence and stability.
Examples
- Neural network training — Minimise training loss with stochastic gradient descent variants (Adam, RMSProp) on large datasets.
- Logistic regression optimisation — Fit classifiers by minimising cross-entropy via batch or mini-batch gradient steps.
- Image deblurring — Solve differentiable inverse problems by optimising reconstruction loss with gradient-based solvers.
Problems
- Choosing a learning rate that's too high (divergence) or too low (slow convergence)
- Getting stuck in local minima or saddle points
- Vanishing or exploding gradients in deep networks
- Sensitivity to feature scaling and initialization
- Noisy or oscillating loss curves with mini-batch variants