Concept - Stacking

Core idea (s.358): use a pool of base classifiers, then use another classifier (the stacker) to combine their predictions for the final decision. Goal: increase the prediction accuracy of a model (s.348; Mlxtend library).

• Level 1: original training data $X$ ($m\times n$) -> models 1..M. Their $M$ predictions become features: $X^{(l2)}$ ($m\times M$) -> Level 2 model -> final prediction ${\hat{y}}^{fin}$ (s.358).
• Example (s.359): Random Forest, SVM, Ridge classifiers -> new features -> L2-Classifier -> labels. Level-1 outputs can be predicted labels or probabilities.
• Demo (s.360): decision regions of RF (blocky), SVC (smooth curves), Ridge (linear) and the Stacking Classifier (a blend of them).

flowchart LR
    X[Raw features] --> RF[Random Forest]
    X --> SVM[SVM]
    X --> R[Ridge]
    RF --> N[New features: predictions]
    SVM --> N
    R --> N
    N --> L2[L2 classifier] --> Y[Labels]

Related

Concept - Majority Voting | Concept - Bagging | Concept - Boosting | Concept - Ensemble Learning

Source chunk: 03 Text Classification - Chunk 12 (s.331-360)