Principal Component Analysis
- Definition
- PCA reduces dimensionality by finding directions (principal components) where data varies most, projecting data into a lower-dimensional space while preserving as much variance as possible. S, picture rotating the data to find the best axes summarizing it. PCA differs from non-linear methods (like t-SNE) by being linear and interpretable, and it's often used as a preprocessing step.
How does it work?
Principal Component Analysis models learn from labeled examples: prepare features, choose a model family, train on examples, and validate on held-out data. Pay attention to data preprocessing, class imbalance, and hyperparameter tuning.
Examples
- Face recognition preprocessing — Reduce dimensionality of image descriptors before nearest-neighbour matching.
- Variance-based feature reduction — Project features to top components to denoise data for downstream models.
- Exploratory data analysis — Visualise high-dimensional datasets on 2–3 principal axes to spot structure.
Problems
- Only captures linear relationships in the data
- Components can be hard to interpret in terms of original features
- Sensitive to feature scaling before applying it
- Can discard information that's useful for the actual downstream task
- Sensitive to outliers skewing the principal directions