Naive Bayes: $h_{NB}=\arg\max_h p(h)\prod_j p(d_j|h)$; attributes independent given the class (s.21-23)

Estimate $p(w|h)=\frac{count(w,h)}{\sum_d count(d,h)}$ from a per-class mega-document; priors from document counts, e.g. $p(like)=300/1000$ (s.27-28)

Confusion matrix TP/FP/FN/TN (s.272-283)

F1 $=\frac{2PR}{P+R}$. Accuracy fails on imbalanced data

ROC: TPR vs FPR over thresholds; AUC = area under it

Multiclass: one-vs-all, or softmax with cross entropy (s.284-291)

Overfitting vs underfitting; complexity drives both (s.331-337)

Boosting: sequential, focus on errors. Stacking: meta-learner (s.353-360)

Sentiment analysis needs more than bag-of-words (s.366-368)

Word sense disambiguation = classification with context features; WordNet synsets (s.369-373)

Getting labels, then validate with inter-annotator agreement (s.374-381)

Bayes rule $p(h|D)=\frac{p(D|h)p(h)}{p(D)}$: posterior = likelihood × prior / evidence (s.13)

Gaussian MLE: $\mu=\frac1m\sum x_i$, $\sigma^2=\frac1m\sum(x_i-\mu)^2$ (s.11)

Curse of dimensionality: $p(d_1..d_n|h)$ needs too much data when $n$ is large (s.20)

Ensembles work via bias-variance; weak vs strong, stable vs unstable learners (s.338-346)

Bagging: bootstrap + majority vote; random forest = bagged trees (s.347-352)

↪ uses → 06 · Decision trees

Learn weights by minimising cross-entropy loss with gradient descent (s.264-269)

Decision boundary is linear (s.270)

Decision tree = function approximation; ID3 builds it greedily (s.301-310)

IG favours many-valued attributes → gain ratio (s.321)

NB independence is given the class, not unconditional

Smoothing with $\alpha=0$ = plain MLE

Accuracy misleads on imbalanced data; use P/R/F1

Multinomial vs Bernoulli vs Gaussian NB (s.146-181)

Spam example (s.61-135): 12 msgs (8 N, 4 S); $p(N)=0.67$, $p(S)=0.33$. "Dear Friend": 0.09 vs 0.01 → normal. "Lunch Money×4": spam = 0 until smoothing

Posterior normalization: divide each score by the sum of scores to get probabilities (s.161)

Gini $=1-\sum p_i^2$, alternative criterion (s.322-327)

03 · Text Classification

Bayes → Naive Bayes → logistic regression → evaluation → trees → ensembles

(s.N) = slide number.

Zero probability / OOV: unseen word zeroes the product (s.29, 119)

Additive smoothing $\frac{count+\alpha}{total+\alpha V}$; $\alpha=1$ = Laplace (s.30, 121-135)

Continuous attributes: threshold splits (s.317-320)

Exam traps (see Past Exam note)

MLE $\arg\max_h P(D|h)$ vs MAP $\arg\max_h P(D|h)P(h)$. Equal when the prior is uniform (s.18-19)

Posterior sharpens as evidence accumulates: flat prior → narrow peak (s.15-17)

Assumption often violated, still works well (text, diagnosis). Ignores word order = bag of words: high bias, low variance (s.21, 136-145)

Log trick: sum of logs avoids underflow (s.204-214)

MNB: word counts / TF-IDF. BNB: binary presence/absence. GNB: continuous, one Gaussian per feature per class

Troll 2 (GNB): log scores ≈ −124 (loves) vs −48 (does not love) → does not love (s.211-238)

Sigmoid $\sigma(z)=\frac1{1+e^{-z}}$ squashes the score into (0,1); gives $p(y=1|x)$ (s.257-263)

Generative (NB) vs discriminative (LR): NB models $p(x|y)p(y)$, LR models $p(y|x)$ directly (s.292-295)

↪ contrasts with → 02 · Naive Bayes for text

Linear regression is a poor classifier: unbounded output, sensitive to imbalanced data (s.253-257)

Precision $\frac{TP}{TP+FP}$; Recall $\frac{TP}{TP+FN}$; trade-off via the threshold

Cross validation can pick which features to keep (s.241-247)

Entropy $H=-\sum p_i\log_2p_i$; information gain $=H(S)-\sum\frac{|S_v|}{|S|}H(S_v)$ (Play Tennis, s.311-316)

Pros/cons (s.328): easy to read but overfits

Pruning: pre (stopping criteria) / post (validation set)

Netflix Prize (s.361-364); No Free Lunch: no model is best for all problems

Likelihood $P(D|h)$ ≠ posterior $P(h|D)$

Compute entropy / IG / Gini by hand on a small table