Linear Discriminant Analysis
- Definition
- Linear Discriminant Analysis (LDA) finds linear combinations of features that separate classes best and models class-conditional Gaussian distributions. Imagine projecting data onto a line where classes become as distinct as possible. LDA differs from SVMs and logistic regression by its probabilistic generative assumptions and closed-form solutions — when those assumptions hold, LDA can be very effective and interpretable.
How does it work?
Linear Discriminant Analysis models learn from labeled examples: prepare features, choose a model family, train on examples, and validate on held-out data. Pay attention to data preprocessing, class imbalance, and hyperparameter tuning.
Examples
- Face recognition projection — Project high-dimensional features to lower dimensions for classification tasks.
- Medical diagnostic scoring — Linear separators for multi-class problems with Gaussian assumptions.
- Feature reduction for classifiers — Reduce dimensionality before applying simple classifiers for speed.
Problems
- Assumes classes share a common covariance structure, often unrealistic
- Sensitive to outliers skewing class means/covariances
- Struggles with non-linearly separable data
- Performance degrades when features are highly correlated or non-Gaussian