Concept - Laplace Smoothing

Concept - Laplace Smoothing (additive smoothing, pseudo-counts, alpha)

Definition (as in the slides): add a number of counts, usually called $\alpha$ (alpha), to every word’s count so that no word probability is ever 0 (s.121-123 in 03 Text Classification - Chunk 05 (s.121-150)). In the example $\alpha =1$, “but we could have set it to anything” (s.122).

Why: if a word never appears in a class (e.g. Lunch in spam), its probability is 0 and the product of word probabilities for that class becomes 0 whatever the other words say. With added counts the score is $>0$ (s.123, 133).

Formula: $p(w\mid c)=\frac{\text{count}(w,c)+\alpha }{N_c+\alpha |V|}$. Example: $p(\text{Lunch}\mid \text{Spam})=\frac{1}{7+4}=0.09$ - 7 words in spam, +4 extra counts (4 vocabulary words $\times \alpha =1$) (s.126-128).

Priors are unchanged: counts added to words do not change the number of messages (8 normal, 4 spam), so $p(N)=0.67$, $p(S)=0.33$ (s.129-130).

Effect on the example: “Lunch Money Money Money Money” gets normal 0.00001 vs spam 0.00122, so it is classified spam (s.131-135).

Related

Concept - Naive Bayes, Concept - Multinomial Naive Bayes, Concept - Bag of Words

Motivation and first step on the slides (03 Text Classification - Chunk 04 (s.91-120))

• Problem (s.107-119): for “Lunch Money Money Money Money”, $p(S)\times p(\text{Lunch}\mid S)\times p(\text{Money}\mid S{)}^4=0$ because Lunch never occurred in spam training data, so every message containing Lunch is classified as normal no matter how often Money appears. See Concept - Zero Probability Problem.
• Fix (s.120): “people usually add 1 count, represented by a black box, to each word in the histograms”: every word in both the normal and the spam histogram gets +1 (spam: Dear 2+1, Friend 1+1, Lunch 0+1, Money 4+1). The recalculated probabilities follow on later slides.

In the spam/ham example (s.154, 157; 03 Text Classification - Chunk 06 (s.151-180))

• Multinomial: $\alpha =1$, denominator = total words in class + $\alpha |V|$ = 6 + 9 = 15; unseen words get $(0+1)/15=1/15$ (s.154).
• Bernoulli: $P(w=1\mid c)=\frac{N_{c,w}+1}{N_c+2}$; with $N_c=2$ the denominator is 4, so an unseen word gets 1/4 = 0.25 (s.157). The “+2” reflects two outcomes (present / absent).