k-Nearest Neighbors
- Definition
- K-Nearest Neighbors (k-NN) classifies new points based on the labels of the k closest training examples. It's simple and nonparametric: store the data and use distance at query time. Think of asking the nearest neighbors what label they have and going with the majority. K-NN differs from models that summarize data into parameters (like SVMs or neural networks) because it keeps raw examples and can be slow at prediction time but very flexible.
How does it work?
k-Nearest Neighbors models learn from labeled examples: prepare features, choose a model family, train on examples, and validate on held-out data. Pay attention to data preprocessing, class imbalance, and hyperparameter tuning.
Examples
- Recommendation by similarity — Nearest-neighbour lookup on user/item embeddings for simple recommender baselines.
- Medical case retrieval — Retrieve past patient cases with similar measurements for clinical decision support.
- Image retrieval — Find images with similar descriptors in a database using k-NN search.
Problems
- Slow inference on large datasets (must compare to many points)
- Sensitive to irrelevant or unscaled features
- Curse of dimensionality degrading distance meaningfulness
- Choosing an appropriate k value and distance metric
- High memory usage storing the full training set