03 Text Classification - Chunk 05 (s.121-150)

Overview

This chunk finishes the StatQuest-style spam-filter example of Concept - Naive Bayes. It shows how additive (Laplace) smoothing with a pseudo-count $\alpha$ removes zero word probabilities, then classifies the message “Lunch Money Money Money Money” as spam (s.121-135). It then explains why Naive Bayes is called naive (it ignores word order and grammar, a bag-of-words view), and why it still works well: high bias, low variance (s.136-145). The last slides move to text data in practice: Concept - Multinomial Naive Bayes (counts / TF-IDF) versus Concept - Bernoulli Naive Bayes (binary presence), with a 4-document worked representation (s.146-150).

Detailed notes

1. Pseudo-counts and the alpha parameter (s.121-123)

• The number of counts added to each word is typically written with the Greek letter $\alpha$ (alpha) (s.121).
• In the example $\alpha =1$, “but we could have set it to anything” (s.122).
• Result: when the probability of observing each word is calculated, we never get 0 (s.123). On the slide the added counts are the black boxes stacked on top of every bar (Dear, Friend, Lunch, Money) in both the normal and spam histograms. See Concept - Laplace Smoothing.

2. Worked smoothed probability: p(Lunch | Spam) (s.124-128)

• Question asked: the probability of seeing Lunch given that it is in spam, written $p(\text{Lunch}\mid \text{Spam})$ (s.124). The slide first highlights only the Lunch bar, then the whole spam histogram (s.125) - the conditional is computed over the spam histogram.
• Numerator: 1 (the original Lunch count in spam is 0, plus the one added count) (s.126).
• Denominator: 7 + 4 = total number of words in the spam messages (7) plus the extra counts added (4, one per vocabulary word with $\alpha =1$) (s.127).
• $p(\text{Lunch}\mid \text{Spam})=\frac{1}{7+4}=0.09$ (s.128).

Added context

The same smoothing gives all the values shown on s.131: spam counts Dear 2, Friend 1, Lunch 0, Money 4 (total 7) become $3/11=0.27$, $2/11=0.18$, $1/11=0.09$, $5/11=0.45$. Normal counts read off the bars (Dear 8, Friend 5, Lunch 3, Money 1, total 17) give $9/21=0.43$, $6/21=0.29$, $4/21=0.19$, $2/21\approx 0.10$. The counts themselves are my reading of the bar heights (the slides only print the final probabilities).

3. Priors are unchanged (s.129-130)

• Adding counts (black boxes) to each word does not change the initial guess that a message is normal, $p(N)=0.67$, or spam, $p(S)=0.33$ (s.129).
• Reason: adding a count to each word did not change the number of messages in the training dataset that are normal (8) or spam (4) (s.130). So $p(N)=8/12=0.67$ and $p(S)=4/12=0.33$.

4. Re-classifying “Lunch Money Money Money Money” (s.131-135)

Smoothed word probabilities (s.131):

Word	p(word | N)	p(word | S)
Dear	0.43	0.27
Friend	0.29	0.18
Lunch	0.19	0.09
Money	0.10	0.45

Priors: $p(N)=0.67$, $p(S)=0.33$.

• Normal score: $p(N)\times p(\text{Lunch}\mid N)\times p(\text{Money}\mid N{)}^4=0.00001$ - “we still get a small number for the normal message” (s.132).
• Spam score: $p(S)\times p(\text{Lunch}\mid S)\times p(\text{Money}\mid S{)}^4=0.00122$ - now a value $>0$ (s.133). (Without smoothing, p(Lunch|S) would be 0, so the spam score would be exactly 0 - that was the problem being fixed; see the earlier chunk.)
• Since the spam value is greater than the normal value (s.134), the message is classified as spam (s.135).

5. Why Naive Bayes is “naive” (s.136-141)

• s.136: “Now let’s talk about why Naive Bayes is naive.”
• The thing that makes it naive is that it treats all word orders the same (s.137).
• Example (normal class): score for “Dear Friend” = $p(N)\times p(\text{Dear}\mid N)\times p(\text{Friend}\mid N)=0.08$ (s.138). Score for “Friend Dear” = $p(N)\times p(\text{Friend}\mid N)\times p(\text{Dear}\mid N)=0.08$ - exactly the same (s.139). Regardless of word order we get 0.08 (s.140), because multiplication is commutative.
• Every language has grammar rules and common phrases, but Naive Bayes ignores all of that (s.141).
• Instead it treats language like a bag full of words, and each message is a random handful of them (s.142) - the Concept - Bag of Words view.
• Why ignore the rules: keeping track of every single reasonable phrase in a language would be impossible (s.143).

6. Despite being naive, it works; bias and variance (s.144-145)

• Even though it is naive, it tends to perform surprisingly well when separating normal messages from spam (s.144).
• In machine-learning lingo: by ignoring relationships among words, Naive Bayes has high bias, but because it works well in practice it has low variance (s.145). See Concept - Bias and Variance.

7. NLP data and the two text variants (s.146-148)

• Preprocessed text is usually represented as: word counts (Bag of Words), frequencies / TF-IDF values, or binary indicators (word present or not) (s.146). See Concept - TF-IDF.
• These are discrete and sparse, not continuous Gaussian-like (s.146).
• For text, Multinomial NB (MNB) or Bernoulli NB (BNB) works well: MNB works well with word counts or TF-IDF; BNB works well with binary word presence/absence (s.146).

Concept - Multinomial Naive Bayes (s.147):

Aspect	MNB
Input expected	Counts of words (Bag of Words), or frequencies / TF-IDF values
Assumption	Each word occurs in a document some number of times, and counts follow a multinomial distribution given the class
Effect	Word frequency matters: if “spam” appears 10 times in an email that is stronger evidence than appearing once
Typical use cases	Spam detection, topic classification, sentiment analysis

Concept - Bernoulli Naive Bayes (s.148):

Aspect	BNB
Input expected	Binary indicators (word present = 1, word absent = 0)
Assumption	Each word is a binary feature
Effect	Only presence/absence matters, not how many times a word appears
Typical use cases	Short texts (tweets, product reviews); when word repetition is less informative

8. Worked representation example (s.149-150)

• s.149 “Example (without normalization)”: spam filter with 4 documents: (1) “buy cheap meds” -> spam; (2) “cheap meds online” -> spam; (3) “meeting schedule today” -> ham; (4) “project schedule update” -> ham. Vocabulary (9 words, in this order): [buy, cheap, meds, online, meeting, schedule, today, project, update].
• s.150 “Representation”:

Doc	Multinomial (counts)	Bernoulli (binary)
Doc1	[1,1,1,0,0,0,0,0,0]	[1,1,1,0,0,0,0,0,0]
Doc2	[0,1,1,1,0,0,0,0,0]	[0,1,1,1,0,0,0,0,0]
Doc3	[0,0,0,0,1,1,1,0,0]	[0,0,0,0,1,1,1,0,0]
Doc4	[0,0,0,0,0,1,0,1,1]	[0,0,0,0,0,1,0,1,1]

The two representations are identical here because no word repeats within a document. “At first glance they look similar, but in longer documents the difference matters”: MNB counts how many times a word appears; BNB only notes if it appeared at least once (s.150).

flowchart TD
    A["Message: Lunch Money Money Money Money"] --> B["Smoothed word probs (alpha = 1)"]
    B --> C["Normal score = p(N) x p(Lunch|N) x p(Money|N)^4 = 0.00001"]
    B --> D["Spam score = p(S) x p(Lunch|S) x p(Money|S)^4 = 0.00122"]
    C --> E{"Compare scores"}
    D --> E
    E -->|"spam larger"| F["Classify as SPAM"]

flowchart TD
    T["Preprocessed text features"] --> W["Word counts / TF-IDF"]
    T --> Bn["Binary present / absent"]
    W --> M["Multinomial NB: frequency matters"]
    Bn --> Br["Bernoulli NB: only presence matters"]

Figures & diagrams

• s.121-123, 126-128 (histograms of word counts): top blue histogram = normal messages (8 messages, $p(N)=0.67$), bottom red histogram = spam (4 messages, $p(S)=0.33$), bars for Dear, Friend, Lunch, Money. Grey/black squares on top of each bar are the added pseudo-counts ($\alpha =1$). Takeaway: every word now has at least one count, so no probability is 0. On s.124-127 the Lunch bar / spam histogram is boxed to show which counts feed $p(\text{Lunch}\mid \text{Spam})$.
• s.128: fraction $1/(7+4)$ with the red ”= 0.09” box; the 7 and 4 are coloured to show the word total and the added counts.
• s.129-130: envelope icons for normal (8, blue) and spam (4, red) with arrows to $p(N)=0.67$, $p(S)=0.33$; takeaway: priors depend only on message counts.
• s.131-135: side panel listing the eight smoothed probabilities and the two score equations (0.00001 vs 0.00122); on s.135 a red spam envelope is attached to the message, i.e. final label = spam.
• s.137-143, 144: envelope icon with the normal word probabilities; formulas for “Dear Friend” vs “Friend Dear” both = 0.08 (red boxes on s.140). s.144 shows both classes’ probability lists with arrows to the “performs surprisingly well” text.
• s.146-148: plain text bullet slides (no figure). s.149: text list of the 4 documents and vocabulary. s.150: two lists of 9-element vectors (counts vs binary).

Key terms

Concept - Naive Bayes, Concept - Laplace Smoothing, Concept - Multinomial Naive Bayes, Concept - Bernoulli Naive Bayes, Concept - Bias and Variance, Concept - Bag of Words, Concept - TF-IDF.

Formulas

• Smoothed word probability (s.126-128): $p(w\mid c)=\frac{\text{count}(w,c)+\alpha }{\text{total words in }c+\alpha \cdot |V|}$, with $\alpha =1$, $|V|=4$: $p(\text{Lunch}\mid S)=\frac{0+1}{7+4}=0.09$.
• Priors: $p(N)=0.67$ (8 messages), $p(S)=0.33$ (4 messages) (s.129-130).
• Score for a message (s.132-133): $\text{score}(c)=p(c){\prod }_{i}p(w_i\mid c)$; e.g. $p(N)p(\text{Lunch}\mid N)p(\text{Money}\mid N{)}^4=0.00001$ and $p(S)p(\text{Lunch}\mid S)p(\text{Money}\mid S{)}^4=0.00122$.
• Order invariance (s.138-140): $p(N)p(\text{Dear}\mid N)p(\text{Friend}\mid N)=p(N)p(\text{Friend}\mid N)p(\text{Dear}\mid N)=0.08$.

Likely exam questions

Style: see Past Exam - Question Bank & Style Guide.

1. Discuss why smoothing (adding $\alpha$) is needed in Naive Bayes. Without it a word never seen in a class gets probability 0, which makes the whole product (score) for that class 0. Adding $\alpha$ counts to every word gives all probabilities $>0$ (s.121-123, 133).
2. Determine p(Lunch | Spam) with $\alpha =1$ as discussed in class. $1/(7+4)=0.09$: 1 = 0+1 added count; 7 = words in spam; 4 = extra counts added (s.126-128).
3. Does adding counts change the priors p(N) and p(S)? Explain. No; they depend on the number of messages (8 normal, 4 spam), which smoothing does not change (s.129-130).
4. Classify “Lunch Money Money Money Money” and show the steps. Normal = 0.00001, spam = 0.00122; spam is larger, so classify as spam (s.131-135).
5. Explain why Naive Bayes is called naive. It treats all word orders the same / ignores grammar and phrases and sees a message as a random handful from a bag of words (s.137, 141-142).
6. Determine the correctness: “Dear Friend” and “Friend Dear” get different scores in Naive Bayes.” False; both get 0.08 (s.138-140).
7. Why does Naive Bayes ignore word relationships? Tracking every reasonable phrase of a language would be impossible (s.143).
8. In ML lingo, what are the bias and variance of Naive Bayes? High bias (ignores relations among words), low variance (works well in practice) (s.145).
9. Compare Multinomial and Bernoulli Naive Bayes.

	MNB	BNB
Input	word counts / frequencies / TF-IDF	binary present/absent
Frequency matters?	Yes (“spam” 10 times > once)	No, only presence
Use cases	spam detection, topic classification, sentiment	short texts (tweets, reviews), repetition uninformative

10. Write the count and binary vectors for “cheap meds online” with vocabulary [buy, cheap, meds, online, meeting, schedule, today, project, update]. $[0,1,1,1,0,0,0,0,0]$ for both (s.149-150).

Common confusions

• Smoothing changes word probabilities, not priors (s.129-130).
• The “+4” in the denominator is $\alpha \times$ vocabulary size (4 words here), not the number of messages (s.127). It is easy to forget when changing $\alpha$ or vocabulary.
• Naive $\ne$ bad: the model ignores order yet performs well; “high bias” does not mean poor accuracy here (s.144-145).
• Multinomial vs Bernoulli: for short non-repeating documents (s.150 example) the vectors are identical; they differ once words repeat (s.150).
• Text features are discrete and sparse, so Gaussian NB is not the natural choice (s.146).

Slide coverage

Slide	Content	Covered in
121	NOTE: added counts per word referred to by alpha	Detailed notes 1
122	In this case alpha = 1, could be anything	Detailed notes 1
123	With added counts we never get 0	Detailed notes 1
124	p(Lunch|Spam) question, Lunch bar highlighted	Detailed notes 2
125	Same, whole spam histogram highlighted	Detailed notes 2
126	Numerator is 1: p(Lunch|Spam)=1/(7+4)	Detailed notes 2
127	Denominator: 7 words in spam + 4 extra counts	Detailed notes 2
128	Result 0.09	Detailed notes 2
129	Counts do not change priors p(N), p(S)	Detailed notes 3
130	Because 8 normal / 4 spam messages unchanged	Detailed notes 3
131	Classify “Lunch Money Money Money Money”; smoothed probabilities listed	Detailed notes 4
132	Normal score = 0.00001	Detailed notes 4
133	Spam score = 0.00122 (>0)	Detailed notes 4
134	Spam value greater than normal	Detailed notes 4
135	Classified as spam	Detailed notes 4
136	Title: why Naive Bayes is naive	Detailed notes 5
137	Treats all word orders the same	Detailed notes 5
138	Score for “Dear Friend” = 0.08	Detailed notes 5
139	Score for “Friend Dear” identical	Detailed notes 5
140	Regardless of order we get 0.08	Detailed notes 5
141	Language has grammar/phrases, NB ignores them	Detailed notes 5
142	Bag of words, random handful	Detailed notes 5
143	Tracking every phrase is impossible	Detailed notes 5
144	Performs surprisingly well separating Normal/Spam	Detailed notes 6
145	High bias, low variance	Detailed notes 6
146	NLP data: counts, TF-IDF, binary; MNB vs BNB	Detailed notes 7
147	Multinomial NB	Detailed notes 7
148	Bernoulli NB	Detailed notes 7
149	Example (without normalization): 4 docs, vocabulary	Detailed notes 8
150	Representation: counts vs binary vectors	Detailed notes 8

Previous: 03 Text Classification - Chunk 04 (s.91-120) | Next: 03 Text Classification - Chunk 06 (s.151-180) | Up: 03 Text Classification - Index