03 Text Classification - Chunk 06 (s.151-180)

Overview

This chunk finishes the worked spam/ham example that compares the two text variants of Naive Bayes: Concept - Multinomial Naive Bayes (MNB, uses word counts) and Concept - Bernoulli Naive Bayes (BNB, uses word presence/absence), both with Concept - Laplace Smoothing (s.151-160). It then shows how to turn the raw, unnormalized scores into proper posterior probabilities (Concept - Posterior Normalization, s.161). The rest of the chunk (s.162-180) starts Concept - Gaussian Naive Bayes (GNB): the assumptions (s.162) and a picture-based “Troll 2” example in which popcorn, soda pop and candy consumption are modelled with one Gaussian per feature per class (s.163-180). The slide numbers 163-180 are taken from a StatQuest-style video (logo top-left) and have almost no text, so the notes below describe each visual step.

Detailed notes

1. Test example: “cheap project meds” (s.151)

Test message: “cheap project meds”; count vector over the 9-word vocabulary: [0,1,1,0,0,0,0,1,0] (s.151).
- Vocabulary order (from s.156): buy, cheap, meds, online, meeting, schedule, today, project, update. So cheap=1, meds=1, project=1.
In Multinomial NB (s.151): “cheap” and “meds” (strong spam indicators) occur once each; “project” (ham indicator) occurs once; prediction likely spam because the frequency of spammy words outweighs “project”.
In Bernoulli NB (s.151): the same words are marked present (1), but repetition would not matter if they occurred multiple times; the prediction “depends more evenly on presence of both spam and ham indicators” -> “could be more balanced”. (Note: the slide’s own numbers on s.156-160 end up giving BNB a stronger spam preference than MNB; see Common confusions.)

2. Multinomial NB worked through (s.152-155)

Step 1 - Training data recap (s.152). Four training documents:

Class	Docs	Word counts	Total words
Spam	1. “buy cheap meds”; 2. “cheap meds online”	buy=1, cheap=2, meds=2, online=1; meeting, schedule, today, project, update = 0	6
Ham	3. “meeting schedule today”; 4. “project schedule update”	meeting=1, schedule=2, today=1, project=1, update=1; buy, cheap, meds, online = 0	6

Step 2 - Priors (s.153). $P (s p am) = 2/4 = 0.5$ , $P (ham) = 2/4 = 0.5$ (class share of training documents).

Step 3 - Likelihoods (s.154). Use Concept - Laplace Smoothing with $α = 1$ . Multinomial formula:

$P (w ∣ c) = \frac{co u n t ( w , c ) + α}{total words in class + α \cdot ∣ V ∣}$

with $∣ V ∣ = 9$ (vocabulary size), so the denominator is $6 + 9 = 15$ .

Word	$P (w ∣ s p am)$	$P (w ∣ ham)$
buy	(1+1)/15 = 2/15	(0+1)/15 = 1/15
cheap	(2+1)/15 = 3/15	1/15
meds	(2+1)/15 = 3/15	1/15
online	(1+1)/15 = 2/15	1/15
meeting	1/15	(1+1)/15 = 2/15
schedule	1/15	(2+1)/15 = 3/15
today	1/15	(1+1)/15 = 2/15
project	1/15	(1+1)/15 = 2/15
update	1/15	(1+1)/15 = 2/15

(On the slide, spam lists buy, cheap, meds, online explicitly and “Others = (0+1)/15 = 1/15”; ham lists meeting, schedule, today, project, update explicitly and “Others = 1/15”. The table above just expands “Others”; the ham values for buy/cheap/meds/online are the slide’s “Others”.)

Step 4 - Classification (s.155).

$P (s p am ∣ x) \propto P (s p am) \cdot P (c h e a p ∣ s p am) \cdot P (m e d s ∣ s p am) \cdot P (p r o j ec t ∣ s p am)$ $= 0.5 \times (3/15) \times (3/15) \times (1/15) = 0.5 \times 9/3375 = 0.5 \times 0.00267 \approx 0.00133$
$P (ham ∣ x) \propto P (ham) \cdot P (c h e a p ∣ ham) \cdot P (m e d s ∣ ham) \cdot P (p r o j ec t ∣ ham)$ $= 0.5 \times (1/15) \times (1/15) \times (2/15) = 0.5 \times 2/3375 = 0.5 \times 0.00059 \approx 0.000296$
Prediction = spam (0.00133 vs 0.000296).

flowchart LR
    A[Training docs<br/>2 spam, 2 ham] --> B[Step 1: word counts per class<br/>6 words each]
    B --> C[Step 2: priors 0.5 / 0.5]
    C --> D[Step 3: Laplace likelihoods<br/>count+1 over 6+9=15]
    D --> E[Step 4: multiply prior x P w given c<br/>for each word in test doc]
    E --> F{Compare scores}
    F -->|0.00133 > 0.000296| G[Predict spam]

3. Bernoulli NB worked through (s.156-159)

1) Vocabulary and binary features (s.156). Vocabulary $V = {$ buy, cheap, meds, online, meeting, schedule, today, project, update $}$ (size 9). BNB uses presence (1) / absence (0) per word. Test doc vector $x = [$ buy=0, cheap=1, meds=1, online=0, meeting=0, schedule=0, today=0, project=1, update=0 $]$ .

2) Class document counts (s.156). $N_{s p am} = 2$ (“buy cheap meds”, “cheap meds online”); $N_{ham} = 2$ (“meeting schedule today”, “project schedule update”).

3) Per-class word presence counts $N_{c, w}$ (s.156) = number of documents of class $c$ containing $w$ :

Spam (over 2 docs): buy=1, cheap=2, meds=2, online=1, meeting=0, schedule=0, today=0, project=0, update=0.
Ham (over 2 docs): buy=0, cheap=0, meds=0, online=0, meeting=1, schedule=2, today=1, project=1, update=1.

4) Laplace-smoothed Bernoulli likelihoods (s.157).

$P (w = 1 ∣ c) = \frac{N _{c, w} + 1}{N _{c} + 2}, N_{c} = 2 \Rightarrow denominator = 4.$

Word	$P (w = 1 ∣ s p am)$	$P (w = 1 ∣ ham)$
buy	2/4 = 0.5	0.25
cheap	3/4 = 0.75	0.25
meds	3/4 = 0.75	0.25
online	0.5	0.25
meeting	0.25	0.5
schedule	0.25	0.75
today	0.25	0.5
project	0.25	0.5
update	0.25	0.5

For absence use $P (w = 0 ∣ c) = 1 - P (w = 1 ∣ c)$ (s.157). Note the “+2” in the denominator (two outcomes: present/absent) versus " $+ α ∣ V ∣$ " in MNB.

5) Class-conditional likelihoods for the test doc (s.158). Bernoulli document likelihood:

$P (x ∣ c) = \prod_{w \in V} P (w = 1 ∣ c)^{x_{w}} [1 - P (w = 1 ∣ c)]^{(1 - x_{w})}$

Present words = {cheap, project, meds}:
- Spam: $(3/4) (1/4) (3/4) = 9/64 = 0.140625$
- Ham: $(1/4) (1/2) (1/4) = 1/32 = 0.03125$
Absent words = {buy, online, meeting, schedule, today, update}:
- Spam: $(1 - 1/2) (1 - 1/2) (1 - 1/4) (1 - 1/4) (1 - 1/4) (1 - 1/4) = (1/2) (1/2) (3/4)^{4} = \frac{1}{4} \times \frac{81}{256} = 81/1024 \approx 0.07910156$
- Ham: the slide first writes $(3/4) (3/4) (1/2) (1/4) (1/2) (1/2)$ and an intermediate value $9/256 \approx 0.03515625$ followed by a ”?”, then says “Careful - apply factors sequentially”: $0.75 \cdot 0.75 = 0.5625$ ; $\times 0.5 = 0.28125$ ; $\times 0.25 = 0.0703125$ ; $\times 0.5 = 0.03515625$ ; $\times 0.5 = 0.017578125 = 9/512$ . The corrected value is 9/512 (the instructor’s self-correction on the slide).

Document likelihoods (s.159):

$P (x ∣ s p am) = \frac{9}{64} \times \frac{81}{1024} = \frac{729}{65536} \approx 0.0111084$
$P (x ∣ ham) = \frac{1}{32} \times \frac{9}{512} = \frac{9}{16384} \approx 0.000549316$

6) Unnormalized posterior scores, including priors (s.159). With equal priors 0.5:

Spam score $= 0.5 \times \frac{729}{65536} = \frac{729}{131072} \approx 0.00555420$
Ham score $= 0.5 \times \frac{9}{16384} = \frac{9}{32768} \approx 0.000274658$
“These are the raw (unnormalized) BNB scores; spam > ham => prediction = spam.”

flowchart TD
    V[Vocabulary of 9 words] --> X[Test vector x: cheap, meds, project = 1, others 0]
    X --> P1[Present words: multiply P w=1 given c]
    X --> P0[Absent words: multiply 1 - P w=1 given c]
    P1 --> L[P x given c = product over ALL 9 words]
    P0 --> L
    L --> S[Multiply by prior 0.5]
    S --> R[Spam 0.00555 vs Ham 0.000275 -> spam]

4. Bottom line: MNB vs BNB (s.160)

Model	Spam score	Ham score	Ratio	Interpretation
Multinomial NB	0.00133	0.000296	$0.00133/0.000296 \approx 4.5$	spam about 4.5x more likely than ham
Bernoulli NB (full formula with absences)	0.00555	0.000275	$0.00555/0.000275 \approx 20.2$	spam about 20x more likely than ham

5. Normalizing raw scores into posteriors (s.161)

Question on slide: “How can we normalize the raw scores into proper posterior probabilities?” Normalization simply means

$P (c ∣ x) = \frac{scor e ( c )}{\sum _{c^{'}} scor e ( c ^{'} )}$

where the denominator is the sum over both classes (Concept - Posterior Normalization).

MNB: raw scores spam 0.00133, ham 0.000296, total 0.001626 -> $P (s p am ∣ x) = 0.00133/0.001626 \approx 0.818$ , $P (ham ∣ x) = 0.000296/0.001626 \approx 0.182$ . MNB confidence: ~82% spam vs 18% ham.
BNB: raw scores spam 0.00555, ham 0.000275, total 0.005825 -> $P (s p am ∣ x) = 0.00555/0.005825 \approx 0.953$ , $P (ham ∣ x) \approx 0.047$ . BNB confidence: ~95% spam vs 5% ham.
Difference: without normalization we only compare magnitudes (ratios ~4.5 vs ~20); with normalization we get actual posterior probabilities. Both models predict spam, but BNB is more confident (95% vs 82%).
Practical remark: normalization is what you use if you want interpretable probabilities (e.g. “spam with 95% certainty”), while unnormalized scores suffice for just choosing the most likely class.

6. Gaussian Naive Bayes - assumptions (s.162)

GNB assumes that features are continuous and follow a normal (Gaussian) distribution given the class.
Works well for sensor readings, continuous measurements, image pixel intensities.
If you use continuous embeddings such as word2vec, GloVe or BERT embeddings (discussed later; see Concept - Word Embeddings), GNB could be applied.

7. GNB intuition - the “Troll 2” example (s.163-180)

Title slide “Gaussian Naive Bayes…” (s.163). The slides (StatQuest “Double BAM!!!” logo, top-left) walk through building the class-conditional Gaussians:

Task (s.164): predict whether someone would love the 1990 movie Troll 2 or not. Two groups of people shown: “Loves Troll 2” (green, 8 figures) and “Does not Love Troll 2” (red, 8 figures).
Data (s.165-166): for the Loves group, a table with columns Popcorn (grams), Soda Pop (ml), Candy (grams): rows 24.3 / 750.7 / 0.2; 28.2 / 533.2 / 50.5; etc. For the Does-not-love group: 2.1 / 120.5 / 90.7; 4.8 / 110.9 / 102.3; etc.
Features (s.167-169): popcorn eaten each day, soda pop drunk, candy eaten (one slide highlights each column in both tables).
Popcorn, Loves Troll 2 (s.170-172): mean = 24, standard deviation = 4; a green Gaussian (bell curve) is drawn (footnote: “Gaussian Distributions are the same as Normal Distributions”).
Popcorn, Does not love (s.173-175): mean = 4, standard deviation = 2; a red bell curve, narrow and tall, located far left of the green curve (low popcorn consumption).
Soda pop (s.176-178): compute mean and standard deviation for people who love Troll 2 (green Gaussian centred high, wide), then the same for those who do not (red Gaussian centred lower, overlapping partially with green). The slides give no numeric mean/SD for soda pop (UNCLEAR whether values were stated; none on slides 176-178).
Candy (s.179-180): green Gaussian for loves: very narrow and tall near zero/low candy; red Gaussian for does-not-love: wide and flat, centred at high candy (about 90-100 g in the table). Both drawn on a third axis labelled Candy.
Takeaway of the build-up: for each class and each continuous feature, estimate a mean and standard deviation from the training data and draw the corresponding Gaussian; the class-conditional likelihood of a new value is read off these curves. (The chunk ends before the slides use these curves to classify a new person; that continues in the next chunk.)

flowchart LR
    D[Training data per class] --> M[For each feature: mean and std dev]
    M --> G[Gaussian curve per feature per class]
    G --> Q[New example: read each feature value on class curves]
    Q --> C[Combine with prior - next chunk]

Figures & diagrams

s.151 Text-only slide: test message, vector [0,1,1,0,0,0,0,1,0], MNB vs BNB commentary.
s.152 Two-column slide: spam docs and ham docs with per-word counts and totals (6 each).
s.153 Two bullets: priors 0.5 / 0.5.
s.154 Laplace-smoothed multinomial formula and per-word likelihood lists for spam (left) and ham (right).
s.155 MNB score computation for both classes ending in “Prediction = spam (0.00133 vs 0.000296)”.
s.156 BNB setup: vocabulary, test presence vector, class doc counts, per-class presence counts.
s.157 BNB Laplace formula $\frac{N _{c, w} + 1}{N _{c} + 2}$ and the two probability lists.
s.158 Bernoulli document likelihood formula; present/absent products, including the visible self-correction (“Careful - apply factors sequentially”).
s.159 Boxed final likelihoods $P (x ∣ s p am)$ , $P (x ∣ ham)$ and unnormalized posteriors.
s.160 “Bottomline” ratio comparison (4.5x vs 20x).
s.161 Normalization formula and both models’ normalized posteriors; difference summary.
s.162 Text: GNB assumptions.
s.163 Title slide “Gaussian Naive Bayes…” with the StatQuest logo.
s.164 Green vs red people icons: loves vs does not love Troll 2.
s.165-166 Tables of popcorn/soda/candy for each group (green and red tables).
s.167-169 Column highlights: popcorn, soda pop, candy.
s.170-172 Popcorn mean/SD and green bell curve for Loves group.
s.173-175 Popcorn mean/SD and red bell curve for Does-not-love group; curves barely overlap.
s.176-178 Soda pop green and red Gaussians; overlapping.
s.179-180 Candy green (narrow) and red (wide) Gaussians; three stacked plots (Popcorn, Soda Pop, Candy) now complete.

Key terms

Concept - Naive Bayes
Concept - Multinomial Naive Bayes
Concept - Bernoulli Naive Bayes
Concept - Gaussian Naive Bayes
Concept - Laplace Smoothing
Concept - Posterior Normalization
Concept - Posterior Probability
Concept - Prior Probability
Concept - Likelihood
Concept - Bag of Words (count vs binary vectors)
Concept - Word Embeddings (continuous features for GNB)

Formulas

Priors: $P (c) = \frac{N _{c}}{N}$ ; example $P (s p am) = P (ham) = 2/4 = 0.5$ (s.153)
Multinomial likelihood with Laplace: $P (w ∣ c) = \frac{co u n t ( w , c ) + α}{total words in class + α ∣ V ∣}$ , $∣ V ∣ = 9$ , denominator 15 (s.154)
MNB score: $P (c ∣ x) \propto P (c) \prod_{w \in x} P (w ∣ c)$ (s.155)
Bernoulli likelihood with Laplace: $P (w = 1 ∣ c) = \frac{N _{c, w} + 1}{N _{c} + 2}$ (s.157); $P (w = 0 ∣ c) = 1 - P (w = 1 ∣ c)$
Bernoulli document likelihood: $P (x ∣ c) = \prod_{w \in V} P (w = 1 ∣ c)^{x_{w}} [1 - P (w = 1 ∣ c)]^{(1 - x_{w})}$ (s.158)
Normalization: $P (c ∣ x) = \frac{scor e ( c )}{\sum _{c^{'}} scor e ( c ^{'} )}$ (s.161)
Worked numbers: MNB 0.00133 vs 0.000296 (ratio 4.5; 0.818 vs 0.182); BNB 0.00555 vs 0.000275 (ratio 20.2; 0.953 vs 0.047) (s.155, 159-161)
Gaussian assumption (s.162): each continuous feature given class ~ $N (μ_{c}, σ_{c}^{2})$ (parameters = per-class mean and standard deviation, s.170-175).

Added context

The Gaussian density $P (x ∣ c) = \frac{1}{2 π σ _{c}^{2}} e^{- (x - μ_{c})^{2} / (2 σ_{c}^{2})}$ is the standard definition of the curve drawn on s.172-180; the slides in this chunk only show the curves, not the formula. Likewise, the expanded per-word table in section 2 simply spells out the slide’s “Others” lines.

Likely exam questions

(Style per Past Exam - Question Bank & Style Guide: short “discuss / describe / explain / determine the correctness” answers.)

Discuss the difference between Multinomial NB and Bernoulli NB. (s.151, 154-158)

	Multinomial NB	Bernoulli NB
Feature	word counts (frequency)	word presence/absence (0/1)
Likelihood used	$P (w ∣ c)$ for each word occurrence in the doc	$P (w = 1 ∣ c)$ for present and $1 - P (w = 1 ∣ c)$ for absent words, over the whole vocabulary
Laplace smoothing	$\frac{co u n t ( w , c ) + α}{total words in class + α ∣ V ∣}$	$\frac{N _{c, w} + 1}{N _{c} + 2}$ ( $N_{c}$ = docs in class)
Repetition	matters	does not matter
Absent words	ignored	explicitly penalized

Classify “cheap project meds” with MNB, as discussed in class. (s.152-155) Priors 0.5/0.5; likelihoods with $α = 1$ , denominator 15; spam $= 0.5 \cdot \frac{3}{15} \cdot \frac{3}{15} \cdot \frac{1}{15} \approx 0.00133$ ; ham $= 0.5 \cdot \frac{1}{15} \cdot \frac{1}{15} \cdot \frac{2}{15} \approx 0.000296$ ; predict spam.
Why do we use Laplace smoothing and what is the denominator in MNB? (s.154) Without it an unseen word gives probability 0 and wipes out the whole product; add $α = 1$ to every count; denominator = total words in class + $α \cdot ∣ V ∣$ = 6 + 9 = 15.
What are the BNB likelihoods $P (w = 1 ∣ s p am)$ for cheap and meeting? (s.157) cheap = (2+1)/4 = 0.75; meeting = (0+1)/4 = 0.25.
Explain the Bernoulli document likelihood formula. (s.158) Product over all vocabulary words: $P (w = 1 ∣ c)$ if present, $1 - P (w = 1 ∣ c)$ if absent.
How can raw NB scores be turned into probabilities? Give the result for MNB and BNB. (s.161) Divide each class score by the sum of the scores over all classes: MNB 0.818 / 0.182, BNB 0.953 / 0.047.
Determine the correctness: “MNB and BNB always give the same confidence.” False; both predict spam but BNB gives ~95% vs ~82% for MNB (s.160-161).
When is normalization needed? (s.161) Only when interpretable probabilities are required; to choose the most likely class the raw scores suffice.
Describe the assumptions of Gaussian Naive Bayes and when it is used. (s.162) Continuous features, normal distribution of each feature given the class; sensor readings, measurements, pixel intensities, continuous embeddings (word2vec, GloVe, BERT).
In the Troll 2 example, what parameters are estimated for each feature? (s.170-175) The mean and standard deviation per feature per class (e.g. popcorn: loves = mean 24, SD 4; does not love = mean 4, SD 2), giving a Gaussian per feature per class.

Common confusions

Count vs presence: MNB counts repeated words; BNB only records 0/1, but BNB also uses absent words in the product (s.151, 158).
Different smoothing denominators: MNB uses total words in class + $α ∣ V ∣$ (15); BNB uses class document count + 2 (4) (s.154, 157).
$N_{c, w}$ counts documents, not word occurrences (s.156), unlike MNB’s $co u n t (w, c)$ (s.152).
Slide 151 vs slides 156-160: s.151 says BNB “could be more balanced”, yet the computed BNB result is more confident in spam (95% vs 82%). Trust the computed numbers (s.160-161).
Slide self-correction: the ham absent-word product on s.158 is first written as 9/256 and then corrected to 9/512 = 0.017578125 - use 9/512 (the final likelihoods on s.159 use 9/512).
Raw scores are not probabilities: they do not sum to 1 until normalized (s.159, 161).
GNB is not for raw word counts: it needs continuous features (e.g. embeddings), unlike MNB/BNB (s.162).

Slide coverage

Slide	Content	Covered in
151	Test example “cheap project meds”, count vector, MNB vs BNB comments	Detailed notes 1
152	MNB step 1: training data recap, word counts	Detailed notes 2
153	MNB step 2: priors	Detailed notes 2
154	MNB step 3: Laplace likelihoods	Detailed notes 2; Formulas
155	MNB step 4: classification -> spam	Detailed notes 2
156	BNB: vocabulary, binary features, class doc counts, presence counts	Detailed notes 3
157	BNB: Laplace-smoothed Bernoulli likelihoods	Detailed notes 3; Formulas
158	BNB: class-conditional likelihoods (present/absent words)	Detailed notes 3; Formulas
159	BNB: document likelihoods and unnormalized posteriors	Detailed notes 3
160	Bottom line: MNB vs BNB ratios	Detailed notes 4
161	Normalizing to posterior probabilities	Detailed notes 5
162	GNB assumptions	Detailed notes 6
163	Title: “Gaussian Naive Bayes…”	Detailed notes 7
164	Troll 2 task: loves vs does not love	Detailed notes 7
165	Data table for people who love Troll 2	Detailed notes 7
166	Data table for people who do not love Troll 2	Detailed notes 7
167	Feature: popcorn	Detailed notes 7
168	Feature: soda pop	Detailed notes 7
169	Feature: candy	Detailed notes 7
170	Popcorn mean (loves) = 24	Detailed notes 7
171	Popcorn standard deviation (loves) = 4	Detailed notes 7
172	Popcorn Gaussian for loves (mean 24, SD 4)	Detailed notes 7; Figures
173	Popcorn mean (does not love) = 4	Detailed notes 7
174	Popcorn standard deviation (does not love) = 2	Detailed notes 7
175	Popcorn Gaussian for does not love	Detailed notes 7; Figures
176	Soda pop mean/SD for loves	Detailed notes 7
177	Soda pop Gaussian for loves	Detailed notes 7; Figures
178	Soda pop Gaussian for does not love	Detailed notes 7; Figures
179	Candy Gaussian for loves	Detailed notes 7; Figures
180	Candy Gaussian for does not love	Detailed notes 7; Figures

Previous: 03 Text Classification - Chunk 05 (s.121-150) | Next: 03 Text Classification - Chunk 07 (s.181-210) | Up: 03 Text Classification - Index

Khaled Notes

Explorer

03 Text Classification - Chunk 06 (s.151-180)

03 Text Classification - Chunk 06 (s.151-180)

Overview

Detailed notes

1. Test example: “cheap project meds” (s.151)

2. Multinomial NB worked through (s.152-155)

3. Bernoulli NB worked through (s.156-159)

4. Bottom line: MNB vs BNB (s.160)

5. Normalizing raw scores into posteriors (s.161)

6. Gaussian Naive Bayes - assumptions (s.162)

7. GNB intuition - the “Troll 2” example (s.163-180)

Figures & diagrams

Key terms

Formulas

Likely exam questions

Common confusions

Slide coverage

Graph View

Table of Contents

Backlinks