Maximum Likelihood Estimation
- Definition
- Maximum Likelihood Estimation (MLE) finds parameter values that make the observed data most probable under a chosen model. You write a likelihood function and optimize it to get point estimates. Think of tuning model knobs until the model would most likely have produced your data. MLE differs from Bayesian methods by not using priors or producing full posterior distributions — it provides a single best-fit answer and is often simpler computationally.
How does it work?
Maximum Likelihood Estimation algorithms typically represent uncertainty explicitly and use probability calculations to combine data and model assumptions. Implementations focus on numerical stability, how to represent distributions (parametric vs sampled), and efficient updates when new data arrive.
Examples
- Logistic regression fitting — Estimate model coefficients by maximising data likelihood for binary outcomes in risk models.
- Gaussian mixture parameters — Fit component means and variances by maximising likelihood with EM for clustering.
- Survival model calibration — Estimate hazard parameters from censored time-to-event data in clinical studies.
Problems
- Overfitting with small sample sizes
- No closed-form solution requiring iterative numerical optimization
- Sensitivity to outliers distorting the estimate
- Local optima in non-convex likelihood surfaces
- Misspecified likelihood model leading to biased parameter estimates