Overfitting
- For Concept - Decision Tree models, overfitting is “one of the most practical difficulty” (s.328).
- s.329 opens the section “Decision Tree: Overfitting”; s.330 “Recap on Overfitting” shows a 2-D scatter (orange vs blue points) with a very irregular, tendril-like decision region wrapped around individual points - the model has fit training noise rather than the general pattern.
Added context
The slides in s.301-330 give no further text on overfitting; the mechanisms (pruning, depth limits) are covered in later chunks. See also Concept - Bias and Variance.
Related
Concept - Decision Tree | Concept - ID3 | Concept - Bias and Variance
Appears in
course: Text & Web Mining deck: 03 Text Classification slides: 332-337 type: concept tags: [midterm, text-web-mining]
Concept - Overfitting
Definition (s.333): small training error but large testing error. A complex model easily overfits the training data and then cannot generalize on testing data.
- In the bias-variance view (s.345-346): overfitting = high variance, low bias.
- Decision trees (s.335): growing the full tree all the time is not a good idea; one noisy sample such as [sunny, hot, normal, true, no] forces the tree to change. General principle: prefer the simplest hypothesis that fits the data.
- How to avoid it in trees: pre-pruning (stopping criteria: min samples for a split, min samples for a leaf, max depth; s.336) and post-pruning (s.337) - see Concept - Decision Tree Pruning.
- Ensembles such as Concept - Bagging reduce variance (s.348).
Related
Concept - Underfitting | Concept - Generalization | Concept - Model Complexity | Concept - Decision Tree Pruning | Concept - Bias-Variance Tradeoff
Source chunk: 03 Text Classification - Chunk 12 (s.331-360)