Gradient Boosting
- Definition
- Gradient Boosting builds a sequence of weak learners (often shallow trees), where each new model focuses on correcting errors made by the previous ensemble. Over many iterations you get a strong predictor. Think of many small trees each learning the residual mistakes and cumulatively improving predictions. It differs from random forests by training sequentially and typically reaching higher accuracy at the cost of more careful tuning and sensitivity to overfitting.
How does it work?
Gradient Boosting models learn from labeled examples: prepare features, choose a model family, train on examples, and validate on held-out data. Pay attention to data preprocessing, class imbalance, and hyperparameter tuning.
Examples
- Credit scoring — Train boosted trees (XGBoost/LightGBM) for high-accuracy risk models in finance.
- Click-through rate prediction — Use gradient-boosted models on sparse features for ad-serving systems.
- Customer lifetime value — Predict long-term value with boosted ensembles that handle heterogeneous features.
Problems
- Prone to overfitting if not carefully regularized (learning rate, depth, early stopping)
- Slower to train sequentially compared to bagging methods
- Sensitive to noisy data and outliers
- Many hyperparameters to tune, making optimization costly
- Can require careful handling of missing values depending on implementation