03 Text Classification - Chunk 04 (s.91-120)

Overview

This chunk finishes the first worked example of Concept - Naive Bayes (spam vs. normal messages) and then exposes its main weakness. The slides (a StatQuest-style “Double BAM” deck; each slide is one animation step) compute the scores for the message “Dear Friend” and conclude it is a normal message (0.09 > 0.01) (s.91-96). A second, “slightly more complicated” message “Lunch Money Money Money Money” is then classified (s.97-120). Intuitively it looks like spam because Money is much more probable in spam, but the spam score becomes exactly 0 because Lunch never appeared in a spam message in the training data. This is the Concept - Zero Probability Problem. The chunk ends by introducing the standard fix: add 1 count (a “black box”) to every word in every histogram, i.e. Concept - Laplace Smoothing (s.119-120).

Detailed notes

1. Setup carried over from earlier slides (visible on every slide s.91-120)

Probabilities estimated from the training data (word histograms) shown on the left of every slide:

	Normal message (N)	Spam (S)
Prior $p(\cdot )$	$p(N)=0.67$	$p(S)=0.33$
$p(\text{Dear}\mid \cdot )$	0.47	0.29
$p(\text{Friend}\mid \cdot )$	0.29	0.14
$p(\text{Lunch}\mid \cdot )$	0.18	0.00
$p(\text{Money}\mid \cdot )$	0.06	0.57

(Blue envelopes = normal messages, red envelopes = spam; s.91-120. The earlier chunk derived these from word counts; see 03 Text Classification - Chunk 03 (s.61-90).)

2. Finishing “Dear Friend” (s.91-96)

• Spam score (s.91): $0.33\times 0.29\times 0.14=0.01$, i.e. $p(S)\times p(\text{Dear}\mid S)\times p(\text{Friend}\mid S)$. “Like before, we can think of 0.01 as the score that Dear Friend gets if it is Spam” (s.91).
• Technically only proportional (s.92): the product is proportional to (symbol $\propto$) the probability that the message is spam given that it says “Dear Friend”: $0.33\times 0.29\times 0.14=0.01\propto p(S\mid \text{Dear Friend})$. It is a score, not a true posterior (the normalising constant $p(\text{Dear Friend})$ is dropped; the same constant would divide both classes).
• Normal score (s.93): earlier result $p(N\mid \text{Dear Friend})\propto 0.09$ (= $0.67\times 0.47\times 0.29$).
• Comparison and decision (s.94-95): the Normal score 0.09 is greater than the Spam score 0.01, so “we will decide that Dear Friend is a Normal Message” (s.95; the slide shows the “Dear Friend” envelope turning blue).
• Recap of the method (s.96): “Now that we understand the basics of how Naive Bayes Classification works…”:
  • $p(N)\times p(\text{Dear}\mid N)\times p(\text{Friend}\mid N)=0.09$
  • $p(S)\times p(\text{Dear}\mid S)\times p(\text{Friend}\mid S)=0.01$
  • Pick the class with the larger score.

3. A more complicated example: “Lunch Money Money Money Money” (s.97-103)

• s.97: transition slide: “…let’s look at a slightly more complicated example.”
• s.98: new message to classify: Lunch Money Money Money Money.
• s.99: NOTE: the message contains the word Money four times. Word counts matter (repeated words are multiplied once per occurrence).
• s.100: intuition: $p(\text{Money}\mid N)=0.06$ versus $p(\text{Money}\mid S)=0.57$. The slide text says “(0.56)” for spam but the formula box shows 0.57 (slide inconsistency, probably rounding; 0.57 is used in the calculations). Money is much more probable in spam than in normal messages.
• s.101: “…then it seems reasonable to predict that this message will end up being spam.” (s.102: “So let’s do the math!“)

4. Normal-message score for the long message (s.103-107)

Built up step by step (each step multiplies one more term):

1. Start with the prior guess $p(N)$ (= 0.67) (s.103).
2. Multiply by the probability of seeing Lunch in a normal message: $p(N)\times p(\text{Lunch}\mid N)$ (0.18) (s.104).
3. Multiply by the probability of seeing Money four times, given normal: $p(N)\times p(\text{Lunch}\mid N)\times p(\text{Money}\mid N{)}^4$ (s.105). The exponent 4 is the word count.
4. Result: $0.67\times 0.18\times 0.06^4=0.000002$ (“this tiny number”) (s.106).

$\text{score}(N)=p(N)\times p(\text{Lunch}\mid N)\times p(\text{Money}\mid N{)}^4=0.000002$

5. Spam score for the long message: it is 0 (s.107-114)

• Same calculation for spam: $p(S)\times p(\text{Lunch}\mid S)\times p(\text{Money}\mid S{)}^4$ (s.107).
• s.108: “…we get 0.”
• s.109: Reason: the probability of seeing Lunch in spam is 0, “since it was not in the Training Data” (the red box highlights $p(\text{Lunch}\mid S)=0.00$).
• s.110-111: plugging in 0 gives $p(S)\times 0\times p(\text{Money}\mid S{)}^4=0$.
• s.112: it doesn’t matter what value we picked for the initial guess $p(S)$ (0.33 highlighted)…
• s.113: …and it doesn’t matter what the probability of Money given spam is (0.57 highlighted)…
• s.114: “…because anything times 0…” (red box around the $\times 0\times$) and s.115: “…is 0.”

6. Why this is a problem (s.116-118)

• s.116: “In other words, if a message contains the word Lunch, it will not be classified as spam” (arrow from the word Lunch to the zero spam score).
• s.117-118 (two identical-looking slides, an animation duplicate): “…and that means we will always classify the messages with Lunch in them as normal, no matter how many times we see the word Money.” The message envelope turns blue; comparison shown: normal $=0.000002$ vs spam $=0$.
• s.119: “And that’s a problem.” (same two scores still on screen).

7. The fix: add one count to each word (s.120)

• s.120: “To work around this problem, people usually add 1 count, represented by a black box, to each word in the histograms.”
• The spam histogram (counts as red boxes) becomes: Dear (2 red + 1 black), Friend (1 red + 1 black), Lunch (0 red + 1 black), Money (4 red + 1 black). The normal histogram (greyed out) also gets a grey box on top of each bar (Dear, Friend, Lunch, Money).
• Effect: Lunch now has a non-zero count in spam, so $p(\text{Lunch}\mid S)>0$ and the product is no longer wiped out. (The recomputed probabilities are on the following slides, i.e. the next chunk.) See Concept - Laplace Smoothing.

Added context

The black-box fix is called Laplace / add-one / additive smoothing, with the added pseudocount usually called $\alpha$ (here $\alpha =1$). With it, $p(w\mid c)=\frac{\text{count}(w,c)+\alpha }{\text{total words in }c+\alpha |V|}$ where $|V|$ is the vocabulary size. The slides in this chunk only show the “add 1 count to each word” step, not this formula; the arithmetic follows in the next chunk. Another practical point not on these slides: multiplying many small probabilities underflows (Concept - Underflow), so implementations sum log-probabilities instead.

Figures & diagrams

• s.91-96 (every slide): left panel with two stacks of envelopes. Blue stack = normal messages ($p(N)=0.67$) with word probabilities Dear 0.47, Friend 0.29, Lunch 0.18, Money 0.06; red stack = spam ($p(S)=0.33$) with Dear 0.29, Friend 0.14, Lunch 0.00, Money 0.57. Faded rows show what is not currently used. Takeaway: class-conditional word probabilities plus the prior are all Naive Bayes needs.
• s.91-95: Spam formula with red box around 0.01 and arrows comparing 0.09 vs 0.01; the “Dear Friend” label turns blue/envelope appears blue when decided as normal (s.95).
• s.96: the two scoring formulas side by side ($=0.09$ and $=0.01$).
• s.98-102: message banner “Lunch Money Money Money Money” and arrows from $p(\text{Money}\mid N)=0.06$ and $p(\text{Money}\mid S)=0.57$: shows the intuitive prediction (spam).
• s.103-106: incremental construction of the normal-class product ending in 0.000002.
• s.107-115: spam-class product ending in 0; red boxes highlight $p(\text{Lunch}\mid S)=0.00$, the $\times 0\times$ term and the result 0.
• s.117-118: both scores displayed (0.000002 vs 0), with the blue “normal” envelope on the message.
• s.120: four histograms (Dear, Friend, Lunch, Money) for normal (light blue/grey) and spam (red); a black box added on top of each bar; Lunch in spam consists only of the black box. Takeaway: smoothing adds a pseudocount so no word has zero probability.

flowchart TD
    M["Message: Lunch Money Money Money Money"] --> N["Normal score<br/>p(N) x p(Lunch|N) x p(Money|N)^4<br/>= 0.67 x 0.18 x 0.06^4 = 0.000002"]
    M --> S["Spam score<br/>p(S) x p(Lunch|S) x p(Money|S)^4<br/>p(Lunch|S) = 0 (never in spam training data)<br/>= 0"]
    N --> D{"Compare scores"}
    S --> D
    D --> R["Classified NORMAL<br/>(wrong intuition: looks like spam)"]
    R --> P["Problem: any unseen word in a class zeroes that class"]
    P --> F["Fix: add 1 count (pseudocount) to every word in every histogram"]

flowchart LR
    A["Dear Friend"] --> B["N: 0.67 x 0.47 x 0.29 = 0.09"]
    A --> C["S: 0.33 x 0.29 x 0.14 = 0.01"]
    B --> D{"0.09 > 0.01"}
    C --> D
    D --> E["Normal message"]

Key terms

• Concept - Naive Bayes
• Concept - Prior Probability
• Concept - Likelihood
• Concept - Zero Probability Problem
• Concept - Laplace Smoothing
• Concept - Underflow (mentioned in the added-context note)
• Related earlier-deck concepts: Concept - Bag of Words (word counts as the features)

Formulas

• Class score for a message with words $w_1..w_n$ (s.96, s.103-107): $\text{score}(c)=p(c){\prod }_{i}p(w_i\mid c)$, with repeated words counted repeatedly, e.g. $p(c)p(\text{Lunch}\mid c)p(\text{Money}\mid c{)}^4$ (s.105, s.107).
• Proportionality (s.92): $p(c){\prod }_{i}p(w_i\mid c)\propto p(c\mid w_1,\dots ,w_n)$
• Decision rule (s.94-95): predict $\arg {\max }_c\text{score}(c)$.
• “Dear Friend”: $0.67\times 0.47\times 0.29\approx 0.09$ (N) vs $0.33\times 0.29\times 0.14\approx 0.01$ (S) (s.91, s.93, s.96).
• “Lunch Money Money Money Money”: N: $0.67\times 0.18\times 0.06^4\approx 0.000002$ (s.106); S: $0.33\times 0\times 0.57^4=0$ (s.108-111).

Likely exam questions

1. Explain how Naive Bayes classifies the message “Dear Friend” as normal or spam. (s.91-96) Compute $p(N)p(\text{Dear}\mid N)p(\text{Friend}\mid N)=0.09$ and $p(S)p(\text{Dear}\mid S)p(\text{Friend}\mid S)=0.01$; the larger score (normal) wins.
2. Why is the score only proportional to $p(S\mid \text{Dear Friend})$? (s.92) The product omits the normalising term (the probability of the message itself), which is identical for both classes so the ranking is unaffected.
3. How are repeated words handled in the calculation? (s.99, s.105) Each occurrence multiplies in once, so Money four times gives $p(\text{Money}\mid c{)}^4$.
4. Discuss the problem that arises when classifying “Lunch Money Money Money Money”. (s.107-118) $p(\text{Lunch}\mid S)=0$ as Lunch never occurred in spam training data; the whole spam score becomes 0 regardless of the prior or of Money’s high probability, so the message is always classified as normal.
5. Why does the prior not matter when the score is 0? (s.112-115) Anything multiplied by 0 is 0.
6. Determine the correctness: “A message containing a word never seen in spam training data can still be classified as spam by (unsmoothed) Naive Bayes.” (s.116-118) False; the spam score is 0, so it is never classified as spam.
7. What is the usual remedy for zero probabilities, as discussed in class? (s.120) Add 1 count (a pseudocount, the “black box”) to each word in each histogram (Laplace/additive smoothing).
8. Which scores are compared in the “Lunch Money” example? (s.117-118) Normal 0.000002 vs spam 0.
9. Why is the tiny number 0.000002 still enough to win? (s.106, s.118) Naive Bayes only compares scores between classes; the absolute magnitude is irrelevant, only that it is larger than the spam score of 0.
10. Describe what the black boxes in the histograms represent. (s.120) One extra count added to every word’s bar in both the normal and spam histograms, so no word has probability 0.

Past-exam link: Naive Bayes / smoothing had no confirmed entry in the past exam; if it appears there, see Past Exam - Question Bank & Style Guide.

Common confusions

• Score vs probability: the numbers 0.09 and 0.01 are scores, proportional to the posteriors, not probabilities that sum to 1 (s.92).
• Intuition vs result: the message looks like spam (lots of Money) but Naive Bayes calls it normal because of the zero for Lunch in spam; the failure is of the estimation (zero count), not of the idea (s.101 vs s.118).
• Zero probability is a training-data artefact: $p(\text{Lunch}\mid S)=0$ does not mean Lunch can never appear in spam, only that it was not observed (s.109).
• Smoothing is applied to counts in every histogram, both classes, not just the zero one (s.120).
• Do not confuse the prior $p(S)=0.33$ with word probabilities; the prior reflects the proportion of messages per class (s.103, s.112).

Slide coverage

Slide	Content	Covered in
91	Spam score for “Dear Friend”: 0.33 x 0.29 x 0.14 = 0.01	Detailed notes 2
92	The 0.01 is proportional to p(S given Dear Friend)	Detailed notes 2
93	Normal score 0.09 highlighted	Detailed notes 2
94	0.09 greater than 0.01	Detailed notes 2
95	Decision: Dear Friend is a normal message	Detailed notes 2
96	Recap of both formulas (0.09 vs 0.01)	Detailed notes 2; Formulas
97	Transition: slightly more complicated example	Detailed notes 3
98	New message: Lunch Money Money Money Money	Detailed notes 3
99	Note: Money occurs four times	Detailed notes 3
100	p(Money given S) 0.57 much higher than p(Money given N) 0.06	Detailed notes 3
101	Reasonable to predict spam	Detailed notes 3
102	”So let’s do the math!”	Detailed notes 3
103	Start with prior p(N)	Detailed notes 4
104	Multiply by p(Lunch given N)	Detailed notes 4
105	Multiply by p(Money given N)^4	Detailed notes 4
106	Normal score = 0.000002	Detailed notes 4
107	Spam formula set up	Detailed notes 5
108	Spam score = 0	Detailed notes 5
109	Because p(Lunch given S) = 0 (not in training data)	Detailed notes 5
110	Plug in 0 for p(Lunch given S)	Detailed notes 5
111	Formula with 0 term = 0	Detailed notes 5
112	Initial guess p(S) does not matter	Detailed notes 5
113	p(Money given S) does not matter	Detailed notes 5
114	Anything times 0…	Detailed notes 5
115	…is 0	Detailed notes 5
116	Message with Lunch will not be classified as spam	Detailed notes 6
117	Always classified as normal regardless of Money count	Detailed notes 6; Figures
118	Same slide as 117 (animation duplicate)	Detailed notes 6
119	”And that’s a problem.”	Detailed notes 6
120	Fix: add 1 count (black box) to each word in histograms	Detailed notes 7; Figures

Previous: 03 Text Classification - Chunk 03 (s.61-90) | Next: 03 Text Classification - Chunk 05 (s.121-150) | Up: 03 Text Classification - Index