Concept - Model Complexity

Slide statements (s.333): a complex model easily overfits the training data; the trained model is then unable to generalize on testing data.

Figure (s.334, source: stackoverflow): error vs model complexity.

• Training error falls continuously as complexity grows.
• Prediction error for new data is U-shaped: it falls, reaches a minimum, then rises.
• The gap between them at high complexity is labelled Optimism.
• Happiness-vs-Wealth example: a straight line (too simple), a smooth saturating curve (about right), a jagged curve through nearly all points (too complex).

flowchart LR
    L[Low complexity] -->|underfitting| M[Medium complexity<br/>minimum test error] -->|overfitting| H[High complexity]

Related

Concept - Overfitting | Concept - Underfitting | Concept - Generalization | Concept - Bias-Variance Tradeoff

Source chunk: 03 Text Classification - Chunk 12 (s.331-360)