Concept - Boosting

Goal: reduce the bias in a model (s.348).

1. Training samples are given weights (initially the same weight) (s.355).
2. At each iteration a new hypothesis is learned.
3. Samples are re-weighted to focus the model on those the most recently learned classifier got wrong.
4. Combine output by voting.

• Algorithms: Gradient Boosting, AdaBoost, XGBoost, LightGBM; Gradient Boosted Decision Trees (s.348, s.355).
• Sequential view (s.356): train a weak model and add it to the ensemble; update the dataset (values or weights) based on the current ensemble results; repeat -> final model.
• Works with Concept - Weak and Strong Learners (weak learners e.g. decision stumps, s.343).
• Figure s.355: Box 1-3 show weak classifiers D1-D3 on re-weighted points; Box 4 combines them.

flowchart LR
    D[Initial dataset] --> W1[Train weak model] --> U1[Re-weight samples] --> W2[Train next weak model] --> U2[...] --> M[Final model]

Related

Concept - Bagging | Concept - Stacking | Concept - Ensemble Learning | Concept - Bias-Variance Tradeoff

Source chunk: 03 Text Classification - Chunk 12 (s.331-360)