03 Text Classification - Chunk 12 (s.331-360)

Overview

This chunk closes the model-evaluation part of the classification lecture and then introduces ensemble learning. It starts with Concept - Generalization (how well a trained model predicts new data), Concept - Model Complexity, and the contrast between Concept - Overfitting and Concept - Underfitting, including how decision trees overfit and how Concept - Decision Tree Pruning (pre-pruning via stopping criteria, post-pruning on a validation set) avoids it (s.331-337). It then motivates Concept - Ensemble Learning (“three kids can defeat a master”, the SQuAD2.0 leaderboard), distinguishes Concept - Weak and Strong Learners and Concept - Stable and Unstable Learners, and explains why ensembles work via the Concept - Bias-Variance Tradeoff (s.338-346). Finally it presents the three common techniques: Concept - Bagging (with Concept - Bootstrapping, Concept - Majority Voting and Concept - Random Forest), Concept - Boosting, and Concept - Stacking (s.347-360).

Detailed notes

1. Generalization (s.331-332)

In ML a model is used to fit the data; once trained it is applied to new data (s.331).
Generalization = the prediction capability of the model on live/new data (s.331).
s.332 “Which model is better?”: two scatter plots of the same SPAM (red) vs Not SPAM (blue) points. Left: a straight line separates most red from blue but misclassifies a few points. Right: a wiggly, S-shaped curve that twists to classify every training point correctly. The slide poses the question without an explicit answer; the implied lesson (developed on s.333-335) is that the simple line is expected to generalize better while the wiggly curve fits noise (overfitting). See Concept - Overfitting.

2. Model complexity, overfitting and underfitting (s.333-334)

A complex model easily overfits the training data; the trained model is then unable to generalize on testing data (s.333).
Definitions (s.333; the slide spells it “underitting”):
- Overfitting: small training error but large testing error.
- Underfitting: large training and testing errors.
s.334 figure (source: stackoverflow). Top row: three scatter plots of Happiness (y) vs Wealth (x) with a fitted dashed curve - (left) a straight line (too simple), (middle) a smooth saturating curve (about right), (right) a jagged curve that goes through nearly every point (too complex). Below: a plot of Model Prediction Error (y) vs Model Complexity (x) with two curves:
- Training Error decreases continuously and flattens as complexity grows.
- Prediction Error for New Data is U-shaped: falls, reaches a minimum in the middle, then rises again at high complexity.
- The gap between the two curves at high complexity is labelled Optimism.
- Three dotted vertical lines link the three fitted curves above to positions on the complexity axis (left = underfit, middle = best, right = overfit).

3. Overfitting for decision trees (s.335)

Question: Is it a good idea to grow the full tree all the time?
- Suppose we add one noisy training sample: [sunny, hot, normal, true, no]. What effect on the earlier tree? (The slide leaves this as a question - this is the PlayTennis-style weather example from the earlier decision tree slides; the noisy example conflicts with the existing pattern so the full tree must grow extra branches to fit it. See Added context below.)
There may exist multiple trees that perform exactly the same on training data - which to select?
- General principle: prefer the simplest hypothesis that fits the data (Occam’s razor idea).

Added context

The effect of the noisy sample is not answered on the slide. It is the standard point: to classify one mislabeled/noisy example correctly, a fully grown tree adds extra splits, making the tree bigger and fitted to noise. The name “Occam’s razor” is added context; the slide only states the principle.

4. How to avoid overfitting in trees (s.336-337)

I. Pre-pruning - stop growing the tree early using stopping criteria (s.336):

Minimum samples for a node split
Minimum samples for a terminal node (leaf)
Maximum depth of tree (vertical depth)
”…” (others, e.g. max_features in the figure)

Figure (source: analyticsvidhya, tree-based modeling tutorial): an example tree with 100 samples at the root. Split on a<1 / a>=1. The a>=1 child is a leaf (label 1, 30 samples). The a<1 child has 70 samples and splits on b<=10 / b>10 into two leaves (label 1 with 30 samples; label 2 with 40 samples). Speech-bubble annotations:

max_features defines the number of features to be considered while deciding each split.
min_samples_split of 70 would not allow any node with less than 70 samples to split.
min_samples_leaf of 30 would not allow any leaf node to have less than 30 samples.
max_depth of 2 would ensure no further splitting even if leaves have the required minimum samples.
Legend: solid circle = tree nodes except terminal; dashed red circle = terminal nodes/leaves; xx = number of samples in a leaf; Y = predicted value of a leaf node; abc = node splitting criteria.

II. Post-pruning (s.337): Grow a “full” tree, then perform post-pruning.

Split data into training and validation set.
Build a full tree that classifies the training data.
Do until further pruning is harmful: greedily remove the node (subtree) whose removal most improves validation-set accuracy.

Figure: accuracy vs number of nodes (x from about 10 to 90). train_accuracy (green) keeps rising from ~0.80 to ~0.98 with more nodes; test_accuracy (blue) stays around 0.76-0.79, peaks near ~55 nodes (red dotted vertical line) and then drops (~0.76-0.77). Takeaway: beyond ~55 nodes more complexity helps only training accuracy, so prune back to around that point. See Concept - Decision Tree Pruning.

5. What is ensemble learning? (s.338-343)

s.338: section divider “What is Ensemble Learning?”.
Intuition (s.339): “Three kids can defeat a master” - cartoon of three children (+ signs) equals one wise master. The right side shows a video thumbnail of a jar of candies (the “wisdom of the crowd” guessing-the-number-of-candies idea; the slide gives no caption - UNCLEAR (slide 339) beyond the picture itself).
Evidence (s.340): SQuAD2.0 leaderboard screenshot (“SQuAD2.0 tests the ability of a system to not only answer reading comprehension questions, but also abstain when presented with a question that cannot be answered based on the provided paragraph”). Columns: Rank, Model, EM (exact match), F1. Words “ensemble” are underlined in red.

Rank	Model	EM	F1
(reference)	Human Performance (Stanford University)	86.831	89.452
1	ALBERT (ensemble model), Google Research & TTIC, Sep 18 2019	89.731	92.215
2	XLNet + DAAF + Verifier (ensemble), PINGAN Omni-Sinitic, Jul 22 2019	88.592	90.859
2	ALBERT (single model), Google Research & TTIC, Sep 16 2019	88.107	90.902
2	UPM (ensemble), Anonymous, Jul 26 2019	88.231	90.713
3	XLNet + SG-Net Verifier (ensemble), Shanghai Jiao Tong University & CloudWalk, Aug 04 2019	88.174	90.702

Takeaway: most top systems are ensembles, and the ensemble version of ALBERT beats its single-model version and even human performance.

Definition (s.341): Ensemble learning = techniques that generate a group of base learners which, when combined, have higher accuracy. Also two distinctions listed: Strong vs. Weak learner; Stable (kNN) vs. Unstable (decision trees, neural networks) ML algorithms. Figure (Polikar, 2008): three classifiers (1, 2, 3), each with its own decision boundary on the same Feature 1 / Feature 2 data, are combined with a summation node (Sigma) into an ensemble-based decision boundary that is smoother/more accurate than any single one.
Stable vs. unstable (s.342, text appears to be pasted from a chatbot answer):
- A stable model (e.g. k-Nearest Neighbors, linear regression, SVM) produces very similar results even if the training set is slightly modified; less sensitive to noise and variation.
- An unstable model (e.g. decision trees and neural networks) is highly sensitive to changes in the training data; small differences can lead to drastically different decision boundaries or predictions.
- Unstable models benefit more from ensemble methods like bagging (e.g. Random Forests) because averaging across learners reduces variance and stabilizes predictions; stable models gain little from bagging because they are already consistent.
- In short: Stable = low variance (e.g. kNN); Unstable = high variance (e.g. decision trees).
- Takeaway: unstable learners (trees, NNs) -> ensembles stabilize them and boost accuracy; stable learners (kNN, linear models) -> only marginal improvements.
- Note: s.341 lists SVM neither as stable nor unstable but s.342 lists SVM as stable; kNN is “stable” on both slides.
Weak vs. strong learners (s.343):
- Weak learner: performs only slightly better than random guessing. Example: a decision stump (1-level decision tree) might reach 55-60% accuracy on a binary task (vs 50% random). Individually weak, but combined in an ensemble they can form a strong learner. This is the basis of Boosting (AdaBoost, Gradient Boosting): train many weak learners sequentially, each correcting the mistakes of the previous one; together they become a strong model.
- Strong learner: already performs well on its own (e.g. random forest, deep neural network, SVM with a good kernel); low error, high accuracy without needing an ensemble. Even strong learners can be ensembled to squeeze out more performance (e.g. stacking multiple strong models in Kaggle competitions).
Summary (s.344):
- Weak learner = slightly better than chance (e.g. decision stump). Boosting turns many weak learners into a strong learner.
- Strong learner = already accurate (e.g. deep neural net). Ensembles can still improve performance, with diminishing returns.
- Ensemble learning works with both, but strategies differ: weak learners -> Boosting; unstable strong learners -> Bagging/Stacking.

6. Bias-variance and why ensembles help (s.345-346)

Bias (s.345): the difference between the average prediction of our model and the correct value we are trying to predict.
Variance (s.345): the variability of the model prediction for a given data point or a value, which tells us the spread of our data (the slide’s wording).
Figure (Bill Howe, UW; src: Domingos 2012): 2x2 grid of bullseye targets. Columns: Low Variance / High Variance. Rows: High Bias / Low Bias. The bull’s-eye is labelled “Truth”.
- High bias + low variance: tight cluster of points far from the centre = Underfitting.
- High bias + high variance: scattered points, away from the centre.
- Low bias + low variance: tight cluster at the centre.
- Low bias + high variance: points scattered around the centre = Overfitting.
Why ensemble? (s.346): (1) Reduce bias, (2) Reduce variance. Prediction error = Bias² + Variance + Irreducible Error.
Figure top: Training Data is split into several samples (colored cylinders), each trains its own model (gear icons), the models’ outputs are combined by Sigma into one final model.
Figure bottom: x = model complexity (i.e. degrees of freedom); y = bias, variance and test error. Bias (orange) falls as complexity increases; variance (blue) rises; test error (green) is U-shaped with its minimum at the optimal balance. Left side = underfitting (high bias), middle = optimal balance, right side = overfitting (high variance).

7. Common ensemble techniques (s.347-348)

s.347 divider “Common Ensemble Techniques”. s.348 summarizes:

Technique	Goal on the slide	Examples on the slide
Bagging	reduce the variance in a model	Random Forest
Boosting	reduce the bias in a model	Ada-Boost, XGBoost, Gradient Boosted Decision Trees
Stacking	increase the prediction accuracy of a model	Mlxtend library (hyperlink)

8. Bagging (s.349-353)

s.349 divider “Bagging”.
s.350 - Bagging = Bootstrap aggregation:
- Train m classifiers from m bootstrap replications (replicates of the training set).
- Combine outputs by voting.
- Decreases error by decreasing the variance.
- Random Forest (randomly select features); ExtraTrees (randomized top-down split) are bagging-style variants (shown in red).
- Figure: Training set -> bootstrap samples T1, T2, …, Tm -> classification models C1, C2, …, Cm -> predictions P1, P2, …, Pm. A “New data” arrow feeds every model; predictions go to Voting -> Final prediction Pf.
s.351 - Bootstrapping:
- Left: initial dataset (full) of 12 numbered items (1-12). An arrow “sampling with replacement” produces bootstrap samples (of size 5), e.g. [1, 10, 3, 8, 3], [4, 2, 11, 9, 3], …, [5, 12, 1, 5, 6]. Note repeated items (3 twice in the first; 5 twice in the last), i.e. an item can appear several times in one sample and some items not at all.
- Right: comparison diagram “Simple random sampling WITH replacement” (the drawn unit is put back into the population so it can be drawn again) vs “Simple random sampling WITHOUT replacement” (the drawn unit is removed from the population).
s.352 - Majority voting:
- Equal: same weight for all models.
- Weighted: the best model gets more weight in a vote.
- Table “Model - Public accuracy score”:

Model	Public accuracy score
GradientBoostingMachine	0.65057
RandomForest Gini	0.75107
RandomForest Entropy	0.75222
ExtraTrees Entropy	0.75524
ExtraTrees Gini (Best)	0.75571
Voting Ensemble (Democracy)	0.75337
Voting Ensemble (3*Best vs. Rest)	0.75667

Takeaway: the equal-weight “democracy” voting ensemble (0.75337) is worse than the best single model (0.75571), but the weighted vote where the best model gets 3 votes versus one vote for each of the rest (0.75667) is better than every single model.

s.353 - Random Forests (source: Towards Data Science): pipeline figure. Initial dataset -> L bootstrap samples (labelled 0, 1, …, L, each with its sample rows) + selected features (a feature subset per tree) -> deep trees fitted on each bootstrap sample and considering only selected features -> random forest (kind of average of the trees).

9. Boosting (s.354-356)

s.354 divider “Boosting”.
s.355 - Boosting idea:
- Training samples are given weights (initially the same weight).
- At each iteration a new hypothesis is learned.
- Training samples are re-weighted to focus the model on samples that the most recently learned classifier got wrong.
- Combine output by voting.
- Examples (red): Gradient Boosting, AdaBoost, XGBoost, LightGBM.
- Figure: three boxes (Box 1 with D1, Box 2 with D2, Box 3 with D3) each show + (blue) and - (red) points with a simple (single-threshold) classifier; points misclassified by one classifier are drawn larger in the next box (more weight; some circled in Box 3). Box 4 combines the three weak classifiers into a more complex classification of the same region.
s.356 - Boosting, sequential view (source: Towards Data Science): legend: ”+ [model icon] = train a weak model and aggregate it to the ensemble model”; circular-arrows icon = “update the training dataset (values or weights) based on the current ensemble model results”. Flow: initial dataset -> train weak model, add to ensemble -> update dataset -> train next weak model, add -> … -> final model.

10. Stacking (s.357-360)

s.357 divider “Stacking”.
s.358 - Core idea: use a pool of base classifiers, then use another classifier (the stacker) to combine their predictions for the final decision.
- Figure: Original training data $X$ (size $m \times n$ ) -> Level 1 models (model 1, model 2, …, model M) -> Level 2 training data: the M predictions $\overset{y}{^}^{μ 1}, \dots, \overset{y}{^}^{μ M}$ are now features forming matrix $X^{(l 2)}$ of size $m \times M$ -> Level 2 model(s) -> Final prediction $\overset{y}{^}^{f in}$ .
s.359 - Stacking example: raw features of a data sample are fed to Random Forest, SVM and Ridge Classifiers; each outputs a column (blue, orange, red). These are concatenated into New Features (a new data sample), which is fed to an L2-Classifier to produce Labels. Note on slide: the level-1 outputs “can be predicted labels or probabilistics” (probabilities).
s.360 - Decision Regions: Demo Case: four 2D decision-region plots for a 3-class data set (classes 0, 1, 2; looks like the Iris data with petal features - not stated on the slide): Random Forest Classifier (axis-aligned rectangular, blocky regions), Support Vector Classifier (smooth curved regions), Ridge Classifier (straight linear boundaries), and Stacking Classifier (a combination of the three: partly blocky, partly curved/linear, with an irregular region). Takeaway: the stacked decision boundary mixes the shapes of its base learners.

Figures & diagrams

s.332 Two scatter plots (SPAM red / Not SPAM blue): straight line vs wiggly S-shaped boundary - simple vs over-complex model.
s.334 Happiness vs Wealth fits (line / smooth / jagged) + training-error vs new-data prediction-error curves against model complexity, “Optimism” gap.
s.336 Example decision tree with max_features, min_samples_split, min_samples_leaf, max_depth annotations and legend.
s.337 Train vs test accuracy against number of nodes; red dotted line at ~55 nodes marks where test accuracy stops improving.
s.339 Cartoon “three kids + = master” and a jar of candies thumbnail (intuition for ensembles).
s.340 SQuAD2.0 leaderboard table (ensembles on top).
s.341 Three classifier decision boundaries combined by Sigma into an ensemble boundary (Polikar 2008).
s.342-344 Text-only explanation slides (stable/unstable, weak/strong, summary).
s.345 Bias/variance bullseye grid (underfitting = high bias, overfitting = high variance).
s.346 Bagging-style schematic of data -> models -> Sigma; bias, variance, test error vs model complexity.
s.350 Bagging flow: bootstrap samples T1..Tm -> models C1..Cm -> predictions -> voting -> Pf.
s.351 Bootstrapping with replacement (items 1-12 -> samples of size 5) and with vs without replacement diagrams.
s.352 Majority voting accuracy table.
s.353 Random forest pipeline (bootstrap samples + selected features).
s.355 Boxes 1-4 boosting illustration (re-weighted +/- points).
s.356 Sequential boosting flow.
s.358 Two-level stacking architecture with $X^{(l 2)}$ .
s.359 Stacking with RF, SVM, Ridge and an L2 classifier.
s.360 Decision regions of RF, SVC, Ridge and Stacking.
Divider slides with no content beyond the title: s.338, 347, 349, 354, 357.

flowchart LR
    A[Model complexity too low] --> U[Underfitting<br/>large train AND test error<br/>high bias]
    B[Model complexity just right] --> G[Good generalization<br/>minimum test error]
    C[Model complexity too high] --> O[Overfitting<br/>small train error, large test error<br/>high variance]
    O --> F1[Pre-pruning: stopping criteria]
    O --> F2[Post-pruning: validation set]
    O --> F3[Ensembles: bagging reduces variance]

flowchart TD
    E[Ensemble Learning<br/>combine base learners] --> BG[Bagging<br/>reduce variance<br/>parallel, bootstrap samples, voting<br/>Random Forest, ExtraTrees]
    E --> BO[Boosting<br/>reduce bias<br/>sequential, re-weight mistakes<br/>AdaBoost, GBDT, XGBoost, LightGBM]
    E --> ST[Stacking<br/>increase accuracy<br/>level-2 classifier on level-1 predictions]

flowchart LR
    X[Training set] --> T1[Bootstrap T1] --> C1[C1] --> P1[P1]
    X --> T2[Bootstrap T2] --> C2[C2] --> P2[P2]
    X --> Tm[Bootstrap Tm] --> Cm[Cm] --> Pm[Pm]
    P1 --> V{Voting}
    P2 --> V
    Pm --> V
    V --> PF[Final prediction Pf]

Comparison of the three techniques:

	Bagging	Boosting	Stacking
Main goal (s.348)	Reduce variance	Reduce bias	Increase prediction accuracy
Training	Learners trained independently on bootstrap samples (s.350)	Learners trained sequentially; samples re-weighted toward previous mistakes (s.355-356)	Level-1 learners’ predictions become features for a level-2 learner (s.358-359)
Combination	Voting (equal or weighted) (s.350, 352)	Voting (s.355)	A trained stacker (L2-classifier) (s.358)
Examples	Random Forest, ExtraTrees	Gradient Boosting, AdaBoost, XGBoost, LightGBM	Mlxtend library

Key terms

Formulas

Prediction error decomposition (s.346): $Prediction Error = Bias^{2} + Variance + Irreducible Error$
Bagging: train $m$ classifiers $C_{1} .. C_{m}$ on $m$ bootstrap samples $T_{1} .. T_{m}$ ; final prediction $P_{f} = vote (P_{1}, \dots, P_{m})$ (s.350).
Stacking: level-2 input matrix $X^{(l 2)} \in R^{m \times M}$ built from the $M$ level-1 predictions $\overset{y}{^}^{μ 1}, \dots, \overset{y}{^}^{μ M}$ ; final prediction $\overset{y}{^}^{f in}$ from the level-2 model (s.358).
Overfitting vs underfitting (s.333): overfitting = small training error, large testing error; underfitting = large training and testing error.

Likely exam questions

Style follows Past Exam - Question Bank & Style Guide (short “discuss / describe / explain” answers, compare tables, lecture wording).

Define generalization. (s.331) The prediction capability of the model on live/new data, after the model has been fitted to training data.
Discuss overfitting and underfitting. (s.333) Overfitting: small training error but large testing error (complex model fits noise, cannot generalize). Underfitting: large training and testing errors (model too simple).
Describe how training error and prediction error on new data change with model complexity. (s.334) Training error keeps decreasing; error on new data falls, reaches a minimum, then rises (U-shape). The gap at high complexity is the optimism.
Is it a good idea to always grow the full decision tree? Explain. (s.335) No. A full tree fits noise (e.g. one noisy sample like [sunny, hot, normal, true, no] forces extra branches) and overfits; prefer the simplest hypothesis that fits the data.
List ways to avoid overfitting in decision trees. (s.336-337) Pre-pruning with stopping criteria (min samples for a split, min samples per leaf, max depth); post-pruning: grow a full tree, then greedily remove nodes that most improve validation accuracy until further pruning is harmful.
Describe the post-pruning procedure. (s.337) Split into training and validation set; build full tree on training data; repeatedly remove the node whose removal most improves validation accuracy; stop when pruning hurts.
What is ensemble learning? (s.341) Techniques that generate a group of base learners which, when combined, have higher accuracy than the individual learners.
Compare stable and unstable learners and say which benefit from ensembles. (s.341-342) Stable (kNN, linear regression, SVM): similar results for slightly modified training data, low variance, marginal gain. Unstable (decision trees, neural networks): sensitive to training data, high variance, gain a lot from ensembles such as bagging.
Compare weak and strong learners. (s.343-344) Weak: slightly better than random guessing (decision stump, 55-60% on binary task); boosting combines many into a strong one. Strong: already accurate (random forest, deep NN, SVM with good kernel); ensembles give diminishing returns; bagging/stacking.
Explain bias and variance and the error decomposition. (s.345-346) Bias = difference between average prediction and true value; variance = variability of prediction for a given point. Error = Bias² + Variance + Irreducible error. High bias = underfitting; high variance = overfitting.
Explain bagging. (s.350-351) Bootstrap aggregation: draw m bootstrap samples (sampling with replacement), train m classifiers, combine by voting; reduces variance. Random Forest also randomly selects features.
Compare bagging, boosting and stacking. (s.348, 350, 355, 358) See comparison table: variance vs bias vs accuracy; parallel vs sequential vs two-level.
Describe boosting. (s.355-356) Samples start with equal weights; each iteration learns a new hypothesis; samples are re-weighted to emphasise those the latest classifier got wrong; outputs combined by voting (AdaBoost, Gradient Boosting, XGBoost, LightGBM).
Describe stacking. (s.358-359) A pool of base classifiers (e.g. random forest, SVM, ridge) produces predictions (labels or probabilities) used as new features for a level-2 classifier which makes the final decision.
Equal vs weighted majority voting? (s.352) Equal: same weight for all models; weighted: best model gets more weight. On the slide’s example, weighted (3*Best vs. Rest) 0.75667 beat the equal-weight democracy 0.75337 and the best single model 0.75571.

Common confusions

Overfitting vs underfitting: overfitting = low training error + high test error (high variance); underfitting = high error on both (high bias) (s.333, s.345).
Bagging vs boosting: bagging targets variance, trains learners independently on bootstrap samples; boosting targets bias, trains sequentially and re-weights misclassified samples (s.348).
Random Forest vs plain bagging: Random Forest = bagging of trees plus random feature selection per tree/split (s.350, s.353); ExtraTrees uses randomized top-down splits (s.350).
Stacking vs voting: voting uses a fixed rule (equal or weighted); stacking trains another classifier (level 2) on the base predictions (s.352 vs s.358).
Sampling with vs without replacement: bootstrapping uses replacement, so a sample can contain repeats (e.g. 3 twice) and omit items (s.351).
Weak vs unstable: weak = slightly above chance (needs boosting); unstable = high-variance but possibly strong (needs bagging/stacking) (s.344).
Pre- vs post-pruning: pre-pruning stops growth by criteria; post-pruning grows the full tree first and then removes nodes using validation accuracy (s.336-337).

Slide coverage

Slide	Content	Covered in
331	Generalization definition	Detailed notes 1
332	Which model is better? line vs wiggly boundary (SPAM)	Detailed notes 1; Figures
333	Model complexity; overfitting vs underfitting	Detailed notes 2
334	Complexity vs error curves, Happiness vs Wealth fits	Detailed notes 2; Figures
335	Overfitting for decision tree, noisy sample, simplest hypothesis	Detailed notes 3
336	Avoid overfitting I: stopping criteria + annotated tree	Detailed notes 4; Figures
337	Avoid overfitting II: post-pruning + accuracy plot	Detailed notes 4; Figures
338	Divider: What is Ensemble Learning?	Detailed notes 5
339	Intuition: three kids can defeat a master	Detailed notes 5; Figures
340	SQuAD2.0 leaderboard	Detailed notes 5; Figures
341	Ensemble learning definition, strong/weak, stable/unstable	Detailed notes 5; Figures
342	Stable vs unstable models explanation	Detailed notes 5
343	Weak and strong learners explanation	Detailed notes 5
344	Summary: weak/strong, boosting vs bagging/stacking	Detailed notes 5
345	Bias-variance definitions + bullseye figure	Detailed notes 6; Figures
346	Why ensemble? reduce bias/variance; error decomposition	Detailed notes 6; Formulas
347	Divider: Common Ensemble Techniques	Detailed notes 7
348	Bagging / Boosting / Stacking summary	Detailed notes 7
349	Divider: Bagging	Detailed notes 8
350	Bagging: bootstrap aggregation, RF, ExtraTrees	Detailed notes 8; Figures
351	Bootstrapping, with/without replacement	Detailed notes 8; Figures
352	Majority voting equal/weighted + accuracy table	Detailed notes 8
353	Random forests pipeline	Detailed notes 8; Figures
354	Divider: Boosting	Detailed notes 9
355	Boosting: re-weighting, algorithms, Box 1-4 figure	Detailed notes 9; Figures
356	Boosting sequential diagram	Detailed notes 9; Figures
357	Divider: Stacking	Detailed notes 10
358	Stacking core idea, two-level architecture	Detailed notes 10; Formulas
359	Stacking with RF, SVM, Ridge -> L2 classifier	Detailed notes 10
360	Decision regions demo: RF, SVC, Ridge, Stacking	Detailed notes 10; Figures

Previous: 03 Text Classification - Chunk 11 (s.301-330) | Next: 03 Text Classification - Chunk 13 (s.361-381) | Up: 03 Text Classification - Index

Khaled Notes

Explorer

03 Text Classification - Chunk 12 (s.331-360)

03 Text Classification - Chunk 12 (s.331-360)

Overview

Detailed notes

1. Generalization (s.331-332)

2. Model complexity, overfitting and underfitting (s.333-334)

3. Overfitting for decision trees (s.335)

4. How to avoid overfitting in trees (s.336-337)

5. What is ensemble learning? (s.338-343)

6. Bias-variance and why ensembles help (s.345-346)

7. Common ensemble techniques (s.347-348)

8. Bagging (s.349-353)

9. Boosting (s.354-356)

10. Stacking (s.357-360)

Figures & diagrams

Key terms

Formulas

Likely exam questions

Common confusions

Slide coverage

Graph View

Table of Contents

Backlinks