03 Text Classification - Chunk 03 (s.61-90)

Overview

This chunk is one continuous worked example of the Concept - Naive Bayes spam filter, told in the “StatQuest” cartoon style (the “Double BAM!!!” logo appears on every slide). Starting from word probabilities already estimated from 12 training messages (8 normal = N, 4 spam = S) over the vocabulary Dear, Friend, Lunch, Money, it replaces the histograms by plain probabilities called Concept - Likelihood values (s.61-64). It then classifies the new message “Dear Friend”: first the Concept - Prior Probability of each class is estimated from the training data ($p(N)=0.67$, $p(S)=0.33$), then multiplied by the likelihood of each word in that class, giving a score of 0.09 for normal and 0.01 for spam (s.65-90). The slides state that each score is only proportional to the Concept - Posterior Probability $p(N\mid \text{Dear Friend})$ (s.79-80). The final comparison/decision happens on the next chunk (the last slide here, s.90, ends with the spam score 0.01).

Detailed notes

1. Replacing histograms by word probabilities (s.61-64)

• Setup carried over from the previous chunk: two piles of training messages, normal (blue envelopes) and spam (red envelopes), each with a histogram of how often the words Dear, Friend, Lunch, Money occur (s.61).
• Because “these histograms are taking up a lot of space”, the histograms are removed but the probabilities are kept (s.61-62).
• Word probabilities per class (s.61-62):

Word	$p(\text{word}\mid N)$ (normal)	$p(\text{word}\mid S)$ (spam)
Dear	0.47	0.29
Friend	0.29	0.14
Lunch	0.18	0.00
Money	0.06	0.57

• Slide 61 animates the transition: faded histograms with arrows pointing from the bars to the numbers. Block counts in the histograms (derived by me, not written on the slide): normal Dear 8, Friend 5, Lunch 3, Money 1 (17 total); spam Dear 2, Friend 1, Lunch 0, Money 4 (7 total) - these reproduce the probabilities (e.g. 8/17 = 0.47, 4/7 = 0.57).
• Note the key contrast: Money is very likely in spam (0.57) but rare in normal messages (0.06); Lunch never occurs in the spam training messages ($p=0.00$) (s.62).
• Terminology Alert (s.63): because these are probabilities of discrete, individual words (not of something continuous like weight or height), the probabilities are also called Likelihoods.
• s.64: Some tutorials call them Probabilities, others Likelihoods; “in this case, the terms are interchangeable. So don’t sweat it.”

2. The new message and the decision problem (s.65-66)

• New message to classify: “Dear Friend” (s.65).
• Goal: “decide if it is a normal message or spam” (s.66). Diagram: the message has two question-mark arrows, one towards the normal pile, one towards the spam pile.

3. Score for “Normal” (s.67-80)

Step A - Prior (s.67-72)

• Start with an initial guess about the probability that any message, regardless of what it says, is a normal message: $p(N)$ (s.67).
• The guess can be any probability we want, but a common guess is estimated from the training data (s.68).
• Computation (s.69): 8 of the 12 messages are normal:

$p(N)=\frac{8}{8+4}=0.67$

• Store it under the normal messages so we don’t forget it (s.70): $p(N)=0.67$.
• Terminology Alert (s.71): the initial guess that we observe a Normal message is called a Prior Probability.

Step B - Multiply by word likelihoods (s.72-78)

• Multiply the prior by the probability that the word Dear occurs in a normal message: $p(N)\times p(\text{Dear}\mid N)$ (s.72).
• …and by the probability that Friend occurs in a normal message: $p(N)\times p(\text{Dear}\mid N)\times p(\text{Friend}\mid N)$ (s.73).
• “Now we just plug in the values that we worked out earlier”: $0.67\times 0.47\times p(\text{Friend}\mid N)$ (s.75), then $0.67\times 0.47\times 0.29$ (s.76), result = 0.09 (s.77).
• s.74 shows the empty formula with $p(N)=0.67$ highlighted in a red box (the first value to be plugged in).

Step C - Interpretation of 0.09 (s.78-80)

• “We can think of 0.09 as the score that Dear Friend gets if it is a Normal Message.” (s.78)
• Technically it is only proportional ($\propto$) to the probability that the message is normal given that it says Dear Friend (s.79):

$0.67\times 0.47\times 0.29=0.09\propto p(N\mid \text{Dear Friend})$

• Put on top of the normal messages “so we don’t forget” (s.80): $p(N\mid \text{Dear Friend})\propto 0.09$.

4. Score for “Spam” (s.81-90)

Exactly the same procedure, with spam probabilities.

• Prior (s.81-83): “just like we did before, we start with an initial guess about the probability that any message, regardless of what it says, is spam”: $p(S)$ (s.81); can be any value, common choice from training data (s.82). Since 4 of the 12 messages are spam (s.83):

$p(S)=\frac{4}{4+8}=0.33$

• Multiply by likelihoods (s.84-89):
  • $p(S)\times p(\text{Dear}\mid S)$ with $p(\text{Dear}\mid S)=0.29$ (s.84),
  • $\times p(\text{Friend}\mid S)$ with $p(\text{Friend}\mid S)=0.14$ (s.85),
  • plug in: $0.33\times p(\text{Dear}\mid S)\times p(\text{Friend}\mid S)$ (s.86-87), $0.33\times 0.29\times p(\text{Friend}\mid S)$ (s.88), $0.33\times 0.29\times 0.14$ (s.89).
• Result (s.90): $0.33\times 0.29\times 0.14=0.01$.

5. Outcome so far (derived comparison)

Class	Prior	$p(\text{Dear}\mid \cdot )$	$p(\text{Friend}\mid \cdot )$	Score
Normal (N)	0.67	0.47	0.29	0.09 (s.77)
Spam (S)	0.33	0.29	0.14	0.01 (s.90)

The slides in this chunk stop at the spam score; the comparison (0.09 > 0.01 so “Dear Friend” is classified as normal) is not shown on s.61-90 and belongs to the next chunk.

Added context

Choosing the class with the larger score is exactly the Naive Bayes decision rule: $\hat{c}=\arg {\max }_{c}p(c){\prod }_{i}p(w_i\mid c)$. The slides here only compute the two scores; the decision rule is not stated on s.61-90. The word-independence assumption behind multiplying the word likelihoods is the “naive” part and is not named on these slides.

flowchart TD
    M["New message: Dear Friend"] --> N["Normal branch"]
    M --> S["Spam branch"]
    N --> N1["Prior p(N) = 8/12 = 0.67"]
    N1 --> N2["x p(Dear|N) = 0.47"]
    N2 --> N3["x p(Friend|N) = 0.29"]
    N3 --> N4["Score = 0.09  (proportional to p(N | Dear Friend))"]
    S --> S1["Prior p(S) = 4/12 = 0.33"]
    S1 --> S2["x p(Dear|S) = 0.29"]
    S2 --> S3["x p(Friend|S) = 0.14"]
    S3 --> S4["Score = 0.01  (proportional to p(S | Dear Friend))"]
    N4 -. compare .-> D["Pick the larger score (next chunk)"]
    S4 -. compare .-> D

Figures & diagrams

• s.61 Two piles of envelopes (blue = normal, red = spam) with faded histograms for Dear/Friend/Lunch/Money and probability labels; arrows indicate the histograms are replaced by the probabilities. Takeaway: keep only the word probabilities.
• s.62-64 Same piles with the 8 word-probability labels; s.63 and s.64 add the “Terminology Alert” text on likelihood vs probability.
• s.65 Text “Dear Friend” with a curved arrow and the label “Now, imagine we got a new message that said:”; piles faded.
• s.66 Darkened piles with blue ”?” and red ”?” arrows from “Dear Friend” to each pile: the classification question.
• s.67-68 Symbol $p(N)$ in the middle; s.68 has arrows from $p(N)$ to both piles (the guess is estimated from the training data of both classes).
• s.69 Fraction $p(N)=\frac{8}{8+4}=0.67$ with blue arrows from the normal pile to the numerator and denominator and a red arrow from the spam pile to the denominator.
• s.70-71 $p(N)=0.67$ written beneath the normal pile; s.71 names it the Prior Probability.
• s.72-77 Step-by-step formula building next to the red-boxed word likelihoods ($p(\text{Dear}\mid N)$ s.72, $p(\text{Friend}\mid N)$ s.73); s.74-77 show the numeric substitution ending in 0.09 (red box).
• s.78-80 Interpretation of 0.09 and the proportionality relation $\propto p(N\mid \text{Dear Friend})$; s.80 places "$p(N\mid \text{Dear Friend})\propto 0.09$" at the top of the slide.
• s.81-83 Mirror of s.67-69 for spam: $p(S)$, then $p(S)=\frac{4}{4+8}=0.33$ with red/blue arrows.
• s.84-90 Mirror of s.72-77 for spam, ending with 0.01 in a red box (s.90).

Key terms

• Concept - Naive Bayes
• Concept - Prior Probability
• Concept - Likelihood
• Concept - Posterior Probability

Formulas

• Prior from training counts (s.69, s.83): $p(N)=\frac{8}{8+4}=0.67$; $p(S)=\frac{4}{4+8}=0.33$
• Score for class N (s.77): $p(N)\times p(\text{Dear}\mid N)\times p(\text{Friend}\mid N)=0.67\times 0.47\times 0.29=0.09$
• Score for class S (s.90): $p(S)\times p(\text{Dear}\mid S)\times p(\text{Friend}\mid S)=0.33\times 0.29\times 0.14=0.01$
• Proportionality (s.79): $0.67\times 0.47\times 0.29=0.09\propto p(N\mid \text{Dear Friend})$

Likely exam questions

1. Why are $p(\text{word}\mid \text{class})$ values also called likelihoods? (s.63-64) They are probabilities of discrete, individual words, not of a continuous quantity (weight, height), so they are also called likelihoods; the two terms are interchangeable here.
2. What is a prior probability and how is it estimated? (s.67-71) The initial guess that a message is of a class regardless of its content; commonly estimated from the training data as the fraction of training messages in that class (8/12 = 0.67 normal, 4/12 = 0.33 spam).
3. Determine the Naive Bayes score of “Dear Friend” for the normal class. (s.72-77) $0.67\times 0.47\times 0.29\approx 0.09$.
4. Determine the score of “Dear Friend” for spam. (s.84-90) $0.33\times 0.29\times 0.14\approx 0.01$.
5. Is 0.09 the probability that the message is normal? Explain. (s.79) No; it is only proportional ($\propto$) to $p(N\mid \text{Dear Friend})$, a score used for comparing the classes.
6. Describe the steps of classifying a new message with this method, as discussed in class. (s.65-90) (1) compute a prior for each class from training counts; (2) look up the likelihood of each word of the message in that class; (3) multiply prior by the word likelihoods to get one score per class; (4) compare scores (decision on the next chunk).
7. Why would “Money” push a message towards spam? (s.62) $p(\text{Money}\mid S)=0.57$ versus $p(\text{Money}\mid N)=0.06$.
8. What problem could $p(\text{Lunch}\mid S)=0.00$ cause? (s.62) Any message containing “Lunch” would get a spam score of 0 regardless of other words. (The slides only show the 0.00 value; the zero-probability problem is not discussed on s.61-90.)
9. Why was the training data’s class ratio used as the prior? (s.68) The prior can be any probability, but a common choice is the estimate from the training data.

Past exam link: compare the style of Past Exam - Question Bank & Style Guide (short “determine / explain” answers); check there whether a Naive Bayes calculation appeared.

Common confusions

• Likelihood vs probability: here interchangeable (s.63-64); do not waste time on the difference.
• Prior vs likelihood: prior $p(N)$ concerns the class only; likelihood $p(\text{word}\mid N)$ concerns a word given the class (s.71, s.63).
• Score vs posterior: 0.09 and 0.01 do not sum to 1, so they are not posteriors; they are proportional to the posteriors (s.79).
• Prior of spam is not 0.5: priors come from the 12 training messages: 8 normal, 4 spam (s.69, s.83).
• Same word, different class: “Dear” has 0.47 under N but 0.29 under S - always use the likelihood of the class being scored (s.72, s.84).
• Rounding: $0.33\times 0.29\times 0.14=0.0134$ is shown as 0.01 on s.90; $0.67\times 0.47\times 0.29=0.0913$ is shown as 0.09 (s.77).

Slide coverage

Slide	Content	Covered in
61	Histograms removed, probabilities kept	Detailed notes 1; Figures
62	Word probabilities per class (N and S)	Detailed notes 1
63	Terminology alert: probabilities = likelihoods	Detailed notes 1; Key terms
64	Terms interchangeable, “don’t sweat it”	Detailed notes 1
65	New message “Dear Friend”	Detailed notes 2
66	Normal or spam? question	Detailed notes 2
67	Initial guess p(N)	Detailed notes 3 (A)
68	Guess estimated from training data	Detailed notes 3 (A)
69	p(N) = 8/(8+4) = 0.67	Detailed notes 3 (A); Formulas
70	p(N)=0.67 stored under normal pile	Detailed notes 3 (A)
71	Terminology alert: Prior Probability	Detailed notes 3 (A); Key terms
72	p(N) x p(Dear given N)	Detailed notes 3 (B)
73	… x p(Friend given N)	Detailed notes 3 (B)
74	Plug in values: start with p(N)=0.67	Detailed notes 3 (B)
75	0.67 x 0.47 x p(Friend given N)	Detailed notes 3 (B)
76	0.67 x 0.47 x 0.29	Detailed notes 3 (B)
77	= 0.09	Detailed notes 3 (B); Formulas
78	0.09 = score of Dear Friend as normal	Detailed notes 3 (C)
79	Proportional to p(N given Dear Friend)	Detailed notes 3 (C); Formulas
80	p(N given Dear Friend) proportional to 0.09 placed on top	Detailed notes 3 (C)
81	Initial guess p(S)	Detailed notes 4
82	Guess from training data	Detailed notes 4
83	p(S) = 4/(4+8) = 0.33	Detailed notes 4; Formulas
84	p(S) x p(Dear given S)	Detailed notes 4
85	… x p(Friend given S)	Detailed notes 4
86	Plug in values: formula with p(S)=0.33	Detailed notes 4
87	0.33 x p(Dear given S) x p(Friend given S)	Detailed notes 4
88	0.33 x 0.29 x p(Friend given S)	Detailed notes 4
89	0.33 x 0.29 x 0.14	Detailed notes 4
90	= 0.01	Detailed notes 4; Formulas

Previous: 03 Text Classification - Chunk 02 (s.31-60) | Next: 03 Text Classification - Chunk 04 (s.91-120) | Up: 03 Text Classification - Index