03 Text Classification - Chunk 02 (s.31-60)

Overview

This part finishes the smoothing discussion for Naive Bayes by handling the Out of Vocabulary (OOV) problem: a test word that never appeared in the training set (s.31). It then shows the scikit-learn implementation MultinomialNB (s.32). From s.33 the lecture works through a step-by-step worked example (taken from a “StatQuest”-style video, mascot logo top-left on s.34-60) of how Naive Bayes learns word probabilities for two classes, normal messages and spam: build a histogram of word counts per class, then turn the counts into conditional probabilities $p(\text{word}\mid \text{class})$. The slides stop right after computing all eight word likelihoods and expose the zero-probability problem for “Lunch” in spam (s.58), which is the motivation for additive smoothing (covered in earlier slides, s.31). The prior probabilities and the final classification of a new message come in the next chunk.

Detailed notes

1. Handling OOV words (s.31)

• Earlier discussion: a word appears in some classes of the training set but not all. This is easily solved with additive smoothing (Concept - Additive Smoothing) (s.31).
• New question: what if a test word is not present in the training dataset at all? This is the Out of Vocabulary (OOV) problem (Concept - Out of Vocabulary (OOV)) (s.31).
• Two ways to handle it (s.31):

Option	How	Remarks
Ignore OOV words	Drop the word and compute the rest	Calculations can still be made without it. If all words were OOV, Naive Bayes reverts to the priors.
Introduce <UNK> token	Replace every OOV word by a new token <UNK>	Vocabulary now has $\Vert V\Vert +1$ tokens. Smoothing must be applied to <UNK> as well.

flowchart TD
    A[Test word not in training vocabulary] --> B{Strategy}
    B -->|Ignore| C[Skip word, use remaining words]
    C --> D[If ALL words OOV: prediction = class priors]
    B -->|UNK token| E[Replace OOV with UNK]
    E --> F[Vocabulary size becomes V+1]
    F --> G[Apply additive smoothing to UNK too]

2. scikit-learn implementation (s.32)

• Screenshot of the documentation of sklearn.naive_bayes.MultinomialNB(alpha=1.0, fit_prior=True, class_prior=None) (s.32).
• Quoted description: “Naive Bayes classifier for multinomial models. The multinomial Naive Bayes classifier is suitable for classification with discrete features (e.g., word counts for text classification). The multinomial distribution normally requires integer feature counts. However, in practice, fractional counts such as tf-idf may also work.” (s.32)
• Parameters visible: alpha=1.0 (the additive/Laplace smoothing strength, default 1), fit_prior=True, class_prior=None (s.32). See Concept - Multinomial Naive Bayes. The meaning of alpha as smoothing is not spelled out on the slide.

Added context

alpha=1.0 is scikit-learn’s additive smoothing parameter ($\alpha =1$ is Laplace smoothing); fit_prior=True means class priors are learned from the training data instead of being uniform. Not stated on the slide.

3. Worked example: learning word probabilities (s.33-60)

• s.33 is a transition slide: “Let’s go through some examples to make sure you got it right!”
• Setup (s.34-35): Imagine we received normal messages from friends and family (blue envelopes, s.34) and also spam (red envelopes), defined on the slide as “unwanted messages that are usually scams or unsolicited advertisements” (s.35).
• The vocabulary in the example has only four words: Dear, Friend, Lunch, Money (s.36).

Step A: histogram of the normal messages (s.36-52)

• First step: make a histogram of all the words that occur in the normal messages (s.36). Counts read from the bar heights (one square = one occurrence): Dear = 8, Friend = 5, Lunch = 3, Money = 1 (s.36-37). Total = 17 words (s.40-41).
• The histogram is used to calculate “the probabilities of seeing each word, given that it was in a normal message” (s.37).
• The notation: $p(\text{Dear})$ is the probability of seeing the word, while $p(\text{Dear}\mid \text{Normal})$ is that probability given that we saw it in a normal message (s.38-39). See Concept - Word Likelihood.
• Calculation for Dear (s.40-43): numerator 8 = total number of times Dear occurred in normal messages (s.40); denominator 17 = total number of words in all of the normal messages (s.41). $p(\text{Dear}\mid \text{Normal})=\frac{8}{17}=0.47$ (s.42). Written compactly over the bar as $p(\text{Dear}\mid N)=0.47$ (“so we don’t forget it”) (s.43).
• Same procedure for Friend (s.44-49): numerator 5 (s.46), denominator 17 (s.47). $p(\text{Friend}\mid \text{Normal})=\frac{5}{17}=0.29$ (s.48-49).
• Lunch (s.50): $p(\text{Lunch}\mid \text{Normal})=\frac{3}{17}=0.18$.
• Money (s.51): $p(\text{Money}\mid \text{Normal})=\frac{1}{17}=0.06$.

Step B: histogram of the spam (s.52-60)

• Now make a histogram of all the words that occur in the spam (s.52). Counts from the red bars: Dear = 2, Friend = 1, Lunch = 0 (no bar), Money = 4 (s.52). Total = 7 words in the spam (s.56).
• Dear (s.53-57): numerator 2 = number of times Dear appeared in the spam (s.55), denominator 7 = total words in the spam (s.56). $p(\text{Dear}\mid \text{Spam})=\frac{2}{7}=0.29$ (s.57), written $p(\text{Dear}\mid S)=0.29$.
• “Likewise, we calculate the probability of seeing the remaining words, given that they were in the spam” (s.58-60):
  • $p(\text{Friend}\mid \text{Spam})=\frac{1}{7}=0.14$ (s.58). Note: the printed fraction line on s.58 reads ”= 0.29” but the small label $p(\text{Friend}\mid S)=0.14$ and the arithmetic $1/7$ give 0.14. UNCLEAR (slide 58): the “0.29” is most likely a typo carried over from the previous slide.
  • $p(\text{Lunch}\mid \text{Spam})=\frac{0}{7}=0$ (s.59). This is the zero-probability problem (Concept - Zero Probability Problem): a word seen in one class but never in the other makes the whole product zero unless smoothing is used (see s.31 and Concept - Additive Smoothing).
  • $p(\text{Money}\mid \text{Spam})=\frac{4}{7}=0.57$ (s.60).

Added context

The slides in this chunk end after the likelihoods. Computing the class priors and the final score $p(N)\prod p(w\mid N)$ vs. $p(S)\prod p(w\mid S)$ for a new message is expected in the following chunk; do not assume numbers for them from this chunk.

flowchart LR
    T[Training messages] --> N[Normal messages]
    T --> S[Spam messages]
    N --> HN[Histogram of word counts<br/>Dear 8, Friend 5, Lunch 3, Money 1<br/>total 17]
    S --> HS[Histogram of word counts<br/>Dear 2, Friend 1, Lunch 0, Money 4<br/>total 7]
    HN --> PN[p word given Normal = count / 17]
    HS --> PS[p word given Spam = count / 7]

Comparison table of the learned likelihoods (derived from s.43, 49-51, 57-60)

Word	Count in Normal	$p(w\mid N)$	Count in Spam	$p(w\mid S)$
Dear	8	8/17 = 0.47	2	2/7 = 0.29
Friend	5	5/17 = 0.29	1	1/7 = 0.14
Lunch	3	3/17 = 0.18	0	0/7 = 0
Money	1	1/17 = 0.06	4	4/7 = 0.57
Total	17	(sums to about 1)	7	(sums to about 1)

Observation (derived): “Money” is rare in normal messages (0.06) but common in spam (0.57), so it is a strong spam indicator; “Lunch” never occurs in spam.

Figures & diagrams

• s.31: text-only slide (OOV).
• s.32: screenshot of the scikit-learn MultinomialNB class documentation (signature and description). Takeaway: Multinomial NB is the standard NB for discrete word counts; tf-idf fractional values may also work.
• s.33: title-only transition slide (“Let’s go through some examples…”).
• s.34: pile of blue envelopes = normal messages from friends and family; mascot logo top-left (present on s.34-60).
• s.35: the blue envelopes now mixed with red envelopes = spam, with arrows labelled “spam (unwanted messages that are usually scams or unsolicited advertisements)”.
• s.36: blue envelopes with arrows pointing to a bar-chart histogram, x-axis words Dear, Friend, Lunch, Money; heights 8, 5, 3, 1; faded red envelopes below (spam, not yet used).
• s.37: same histogram, arrow to the “Lunch” bar: use the histogram to get the probability of each word given a normal message.
• s.38: “Dear” bar highlighted with a red box, label $p(\text{Dear}$ (the unconditioned probability being introduced).
• s.39: the whole histogram boxed in red, label $p(\text{Dear}\mid \text{Normal})$: “given that we saw it in a normal message”.
• s.40: Dear bar boxed; $p=\frac{8}{17}$ with 17 greyed: numerator is the count of Dear.
• s.41: whole histogram boxed; arrow to denominator 17 = total words in normal messages.
• s.42: result boxed in red: $=0.47$.
• s.43: result placed over the Dear bar as $p(\text{Dear}\mid N)=0.47$.
• s.44-45: same steps for Friend: bar highlighted and fraction $\frac{5}{17}=0.29$ greyed (s.44), whole histogram boxed (s.45).
• s.46: Friend bar boxed, numerator 5 explained.
• s.47: whole histogram boxed, denominator 17 explained.
• s.48: result 0.29 boxed.
• s.49: $p(\text{Friend}\mid N)=0.29$ placed over the Friend bar.
• s.50: Lunch bar with label $p(\text{Lunch}\mid N)=0.18$ ($3/17$).
• s.51: Money bar with label $p(\text{Money}\mid N)=0.06$ ($1/17$); all four normal probabilities are now stacked above the bars.
• s.52: new red histogram below the (faded) blue one for the spam messages: Dear 2 squares, Friend 1, Lunch 0, Money 4.
• s.53: Dear (spam) bar boxed; $\frac{2}{7}=0.29$ greyed.
• s.54: whole spam histogram boxed, label $p(\text{Dear}\mid \text{Spam})$: “given that we saw it in the spam”.
• s.55: Dear bar boxed; numerator 2.
• s.56: whole spam histogram boxed; denominator 7.
• s.57: result 0.29 boxed and placed as $p(\text{Dear}\mid S)=0.29$.
• s.58: Friend: $p(\text{Friend}\mid S)=0.14$ (see typo remark above).
• s.59: Lunch: $p(\text{Lunch}\mid S)=0$ ($0/7$), no bar.
• s.60: Money: $p(\text{Money}\mid S)=0.57$ ($4/7$). Takeaway: normal messages favour Dear/Friend/Lunch, spam favours Money.

Key terms

• Concept - Out of Vocabulary (OOV) (s.31), including the <UNK> token

• Concept - Additive Smoothing (s.31)

• Concept - Multinomial Naive Bayes (s.32)

• Concept - Word Likelihood (s.37-60)

• Concept - Zero Probability Problem (s.59)

• Concept - Naive Bayes Classifier (s.31, 33-60)

• Concept - Bag of Words (the word-count histograms ignore word order; s.36, 52)

• Concept - Naive Bayes (general NB concept; see also Concept - Prior Probability, Concept - Likelihood)

Formulas

• Word likelihood by counting (s.40-42, 53-57): $p(w\mid c)=\frac{\text{count}(w,c)}{{\sum }_{w^{\prime }}\text{count}(w^{\prime },c)}$ where the denominator is the total number of words in all messages of class $c$ (17 for Normal, 7 for Spam).
• Examples: $p(\text{Dear}\mid N)=\frac{8}{17}=0.47$; $p(\text{Friend}\mid N)=\frac{5}{17}=0.29$; $p(\text{Lunch}\mid N)=\frac{3}{17}=0.18$; $p(\text{Money}\mid N)=\frac{1}{17}=0.06$ (s.42, 48, 50, 51).
• Spam: $p(\text{Dear}\mid S)=\frac{2}{7}=0.29$; $p(\text{Friend}\mid S)=\frac{1}{7}=0.14$; $p(\text{Lunch}\mid S)=\frac{0}{7}=0$; $p(\text{Money}\mid S)=\frac{4}{7}=0.57$ (s.57-60).
• Vocabulary with unknown token: $\Vert V\Vert +1$ tokens (s.31).

Likely exam questions

1. What is the OOV problem and how can it be handled? A test word that does not appear in the training dataset (s.31). Either ignore it, or replace all OOV words with an <UNK> token (vocabulary becomes $\Vert V\Vert +1$, and <UNK> must also be smoothed) (s.31).
2. What happens in Naive Bayes if all words of a test document are OOV and OOV words are ignored? NB reverts to the class priors (s.31).
3. Discuss the difference between a word missing from some classes and an OOV word. The first is a word seen in training in some classes but not all, solved by additive smoothing; the second is absent from the entire training set (s.31).
4. Compute $p(\text{Dear}\mid \text{Normal})$ given Dear occurs 8 times and the normal messages contain 17 words in total. $8/17=0.47$ (s.42).
5. Explain how the word probabilities are estimated for each class in Naive Bayes. Build a histogram of word counts for each class, divide each count by the total number of words in that class (s.36-41, 52-56).
6. Why is $p(\text{Lunch}\mid \text{Spam})=0$ a problem? Any message containing “Lunch” would get probability 0 for spam regardless of other words, because the likelihoods are multiplied; fix with additive smoothing (s.59, s.31).
7. Which variant of Naive Bayes suits word counts, and what does scikit-learn’s alpha default to? Multinomial NB; alpha=1.0 (s.32).
8. Can tf-idf values be used with MultinomialNB? The multinomial distribution normally needs integer counts, but fractional counts such as tf-idf may also work in practice (s.32).
9. Which word is the best spam indicator in the example and why? “Money”: 0.06 in normal vs 0.57 in spam (s.51, 60).
10. Determine the correctness: “In the example, the denominator for spam probabilities is 17.” False, it is 7 (total words in spam, s.56); 17 is for the normal messages (s.41).

See also Past Exam - Question Bank & Style Guide for the question style (this specific example has not been mapped to a past exam question yet).

Common confusions

• $p(\text{word})$ vs $p(\text{word}\mid \text{class})$: the example conditions on the class (“given that we saw it in a normal message”) (s.38-39).
• Different denominators: each class is normalised by its own total word count (17 vs 7), not by the vocabulary size or the number of messages (s.41, 56).
• Counts are word occurrences, not documents: each square on the histogram is one occurrence of the word (s.36, 52).
• Ignoring vs <UNK>: ignoring an OOV word leaves vocabulary unchanged; <UNK> adds one token and needs its own smoothing (s.31).
• Zero count vs OOV: Lunch has zero count in spam but is in the vocabulary (seen in normal messages), so additive smoothing fixes it; an OOV word is not in the vocabulary at all (s.31, 59).
• Slide typo: s.58 shows ”= 0.29” next to $1/7$; the correct value is 0.14.

Slide coverage

Slide	Content	Covered in section
31	Handling OOV: ignore vs <UNK>, vocabulary $\Vert V\Vert +1$	Detailed notes 1
32	scikit-learn MultinomialNB documentation screenshot	Detailed notes 2
33	Transition: “Let’s go through some examples”	Detailed notes 3
34	Normal messages (blue envelopes)	Detailed notes 3 (setup)
35	Spam messages (red envelopes), definition	Detailed notes 3 (setup)
36	Histogram of normal-message words (Dear, Friend, Lunch, Money)	Step A
37	Use histogram to compute word probabilities given normal	Step A
38	Introducing $p(\text{Dear})$	Step A
39	$p(\text{Dear}\mid \text{Normal})$ notation	Step A
40	Numerator 8	Step A
41	Denominator 17	Step A
42	Result 0.47	Step A
43	$p(\text{Dear}\mid N)=0.47$ stored over bar	Step A
44	Friend: probability introduced	Step A
45	Friend given normal message	Step A
46	Friend numerator 5	Step A
47	Friend denominator 17	Step A
48	Friend result 0.29	Step A
49	Friend stored over bar	Step A
50	Lunch = 3/17 = 0.18	Step A
51	Money = 1/17 = 0.06	Step A
52	Histogram of spam words	Step B
53	Dear in spam introduced	Step B
54	Dear given spam	Step B
55	Dear numerator 2	Step B
56	Denominator 7	Step B
57	Dear result 0.29 stored	Step B
58	Friend in spam = 1/7 = 0.14 (typo 0.29 on slide)	Step B
59	Lunch in spam = 0/7 = 0	Step B, Zero probability
60	Money in spam = 4/7 = 0.57	Step B

Previous: 03 Text Classification - Chunk 01 (s.1-30) | Next: 03 Text Classification - Chunk 03 (s.61-90) | Up: 03 Text Classification - Index