03 Text Classification - Chunk 08 (s.211-240)

Overview

This chunk is the end of the worked Gaussian Concept - Naive Bayes example (a StatQuest-style “Troll 2” example with three features: popcorn, soda pop, candy). It starts by explaining that multiplying many small likelihoods causes underflow, so the classifier works with the log of the score instead: the log turns the product into a sum of logs (s.211-214). The log score for Loves Troll 2 is computed step by step as -124 (s.215-222). The same procedure is then repeated for Does Not Love Troll 2, giving about -48 (-47.66) (s.223-236). Because -48 > -124, the person is classified as Does Not Love Troll 2 (s.237-238). The chunk closes with a note that, on the raw data, the person looks like a “Loves Troll 2” person because of the large popcorn intake (s.239-240) - which sets up a discussion of the naive Bayes assumption/weakness in the next chunk.

Slide text is mostly step-by-step animation builds: many consecutive slides show the same layout with one more number revealed. All 30 slides were viewed as images (manifest text is empty for every slide).

Detailed notes

1. Why the log: avoiding underflow (s.211-214)

• Context shown on slide: the Loves Troll 2 Score is the product $0.5\times 0.06\times 0.004\times \text{(a really, really small number)}$ (s.211). The multiplicands are the prior (0.5) and the likelihood values read off the three Gaussian curves (Popcorn, Soda Pop, Candy) (s.211).
• Statement: “So we use the log() function to avoid Underflow” (s.211). See Concept - Underflow.
• Note on slide: any log will do, but the natural log (ln), i.e. log base $e$, is the most commonly used log in statistics and machine learning (s.211).
• Step 1: “we take the log of everything”: $\log (0.5\times 0.06\times 0.004\times \text{tiny})$ (s.212).
• Step 2: the log turns the multiplications (the red-boxed “x” signs) … (s.213)
• Step 3: … into the sum of individual logs: $\log (0.5)+\log (0.06)+\log (0.004)+\log (\text{tiny})$ (s.214).

2. Log score for “Loves Troll 2” (s.215-222)

All logs are base $e$ (s.215).

Term	Value of log	Slide
$\log (0.5)$	-0.69	s.215-216
$\log (0.06)$	-2.8	s.217
$\log (0.004)$	-5.5	s.218
$\log (\text{really, really small number})$	-115	s.219
Sum	-124	s.220-221

• “Now we just add this up…” (red box around the four values) (s.220), “…and we get -124” (s.221).
• Conclusion: $\log (\text{Loves Troll 2 Score})=-124$ (s.222). Note that the log of the tiny number (-115) is only a moderately large negative number, so no underflow occurs, whereas the raw product would be vanishingly small (s.219 vs s.211).

3. Score for “Does Not Love Troll 2” (s.223-236)

• Start: “Now let’s calculate the score for Not Loving Troll 2” (s.223). On the right the same three feature plots are shown with the red curves (No Love) highlighted and the green curves (Love) faded; the person’s data are the X marks on each plot (s.223).
• Initial guess (prior): $p(\text{No Love Troll 2})=0.5$, i.e. $p(\text{No Love})$ (s.224).
• Multiply by the likelihood of each observed feature value given No Love (s.225-227):
  • popcorn: $L(\text{popcorn}=20\mid \text{No Love})$ - 20 grams of popcorn (s.225)
  • soda pop: $L(\text{soda pop}=500\mid \text{No Love})$ - 500 mls (s.226)
  • candy: $L(\text{candy}=25\mid \text{No Love})$ - 25 grams (s.227)
• Full score formula:

$\text{Does not Love Troll 2 Score}=p(\text{No Love})\times L(\text{popcorn}=20\mid \text{No Love})\times L(\text{soda pop}=500\mid \text{No Love})\times L(\text{candy}=25\mid \text{No Love})$

• “So let’s plug in the numbers…” (s.228): prior = 0.5; popcorn likelihood = “a tiny number” (s.229) (exact value not written on the slide); soda pop likelihood = 0.000079 (s.230); candy likelihood = 0.02 (s.231).
• Take the log of everything (s.231), which turns the multiplication into a sum (s.232) of logs (s.233): $\log (0.5)+\log (\text{tiny})+\log (0.000079)+\log (0.02)$.
• “Now we just do the math…” (s.234): $-0.69+-33.6+-9.45+-3.91$ (s.235). Reading: -0.69 = log of prior; -33.6 = log of popcorn likelihood; -9.45 = log(0.000079); -3.91 = log(0.02).
• Sum = -48 (s.236) (slide s.238 shows the more precise value -47.66).

4. Classification decision (s.237-238)

• $\log (\text{Loves Troll 2 Score})=-124$ vs $\log (\text{Does not Love Troll 2 Score})=-48$ (s.237).
• “Since the score for Does Not Love Troll 2 (-48) is greater than the score for Loves Troll 2 (-124)…” (s.237) -> “We will classify this person as someone who Does Not Love Troll 2” (s.238). Rule: pick the class with the highest (least negative) log score.
• Mermaid summary of the decision is in Figures & diagrams.

5. Caveat: the raw data seem to say “Loves” (s.239-240)

• NOTE: “When we look at the raw data, it almost looks like we should have classified this person as someone who Loves Troll 2” (s.239). The plots show all three features with both red (No Love) and green (Love) curves, the person (X marks) and the person icon in red (s.239).
• Reason given: “After all, they ate a lot more popcorn than the average person who Does Not Love Troll 2…” - the X for popcorn lies far out in the tail of the red curve, closer to the green (Love) curve (s.240). The sentence continues in the next chunk (UNCLEAR whether the explanation continues in s.241; this chunk ends mid-sentence with ”…”).

Figures & diagrams

• s.211-222 (left side): tower of numbers $0.5\times 0.06\times 0.004\times \text{tiny}$, transformed into the sum of logs with red boxes marking the “x” operators that become ”+” (s.213-214); a faded group of 8 green person icons labelled “Loves Troll 2 Score”, with $p(\text{Loves Troll 2})=0.5$ at the top (s.211-221). At s.222 the heading becomes $\log (\text{Loves Troll 2 Score})=-124$.
• s.211-222 (right side, faded): three stacked Gaussian plots (Popcorn, Soda Pop, Candy) with green (Loves) curves in front, X marks for the person’s values and a grey arrow pointing to the left of each x-axis (illustrating a very small likelihood far out in the tail for the “Loves” curve). Takeaway: the likelihoods read off the green curves are small, especially candy (the green candy curve is a narrow peak far from the X).
• s.223-236: same three plots with the red “No Love” curves highlighted one at a time (popcorn s.225/229, soda pop s.226/230, candy s.227/231). Popcorn: red curve is a narrow peak at low grams, the X is far in the right tail -> tiny likelihood; soda pop: wide red curve, X in right tail but not as far; candy: low flat wide red curve, X near its peak region -> moderate likelihood (0.02).
• s.223-236 (left): 8 red person icons = “Does not Love Troll 2” training group; the score expression builds line by line (prior, three likelihoods).
• s.237-238: side-by-side comparison of the two groups (8 green icons, 8 red icons), the two log scores with red boxes, and an arrow to the grey person who is then classified.
• s.239-240: the green and red curves all fully drawn with X marks; the person icon is coloured red (classified as No Love). At s.240 the popcorn plot is highlighted and an arrow points to the X in the popcorn plot.

flowchart TD
    A[New person: popcorn=20, soda pop=500, candy=25] --> B[Loves Troll 2 score]
    A --> C[Does Not Love Troll 2 score]
    B --> B1["log p(Love) + sum of log likelihoods"]
    C --> C1["log p(No Love) + sum of log likelihoods"]
    B1 --> D["log score = -124"]
    C1 --> E["log score = -48 (-47.66)"]
    D --> F{Compare}
    E --> F
    F -->|"-48 > -124"| G[Classify: Does Not Love Troll 2]

flowchart LR
    P["product: 0.5 x 0.06 x 0.004 x tiny"] -->|"take log of everything"| L["log(product)"]
    L -->|"log turns x into +"| S["log 0.5 + log 0.06 + log 0.004 + log tiny"]
    S -->|"add up"| R["-0.69 - 2.8 - 5.5 - 115 = -124"]

Key terms

• Concept - Naive Bayes - classifier whose score = prior x product of feature likelihoods.
• Concept - Gaussian Naive Bayes - likelihoods read from Gaussian (normal) curves, one per feature per class (Popcorn, Soda Pop, Candy plots).
• Concept - Underflow - why we take $\log$ of the score (the “log trick”).
• Concept - Likelihood - $L(x\mid \text{class})$ read off the curves.
• Concept - Prior Probability - the initial guess $p(\text{class})=0.5$.

Formulas

• Score for a class $c$ (as used on the slides): $\text{Score}(c)=p(c){\prod }_{j}L(x_j\mid c)$ (s.211, s.225-228).
• Log trick: $\log (p(c){\prod }_{j}L(x_j\mid c))=\log p(c)+{\sum }_j\log L(x_j\mid c)$ (s.212-214, s.231-233).
• Natural log: base $e$, written ln or log() (s.211).
• Worked values: $\ln 0.5=-0.69$; $\ln 0.06=-2.8$; $\ln 0.004=-5.5$; log of tiny number = -115; sum = -124 (s.215-221). For “No Love”: $-0.69+-33.6+-9.45+-3.91=-48$ (-47.66) (s.235-236, s.238).
• Decision rule: predict the class with the larger log score (-48 > -124 -> Does Not Love Troll 2) (s.237-238).

Likely exam questions

Style follows Past Exam - Question Bank & Style Guide (short “discuss / explain / determine correctness” answers).

1. Why do we take the log of the naive Bayes score? (as discussed in class) - The score is a product of many small numbers (prior x likelihoods); the product can become so small the computer cannot represent it (underflow). The log turns the product into a sum of logs, which are moderate negative numbers (s.211-214).
2. What does the log do to multiplication? - It turns multiplication into addition: $\log (a\times b)=\log a+\log b$ (s.213-214).
3. Which log base is typically used? - Any base works; the natural log (base $e$) is the most commonly used in statistics and machine learning (s.211).
4. Compute/explain how the Loves Troll 2 log score is obtained. - Sum $\log (0.5)+\log (0.06)+\log (0.004)+\log (\text{tiny})=-0.69-2.8-5.5-115=-124$ (s.215-221).
5. Compute/explain the Does-Not-Love score. - Prior 0.5, then likelihoods for popcorn=20, soda pop=500 (0.000079), candy=25 (0.02); log sum = -0.69 - 33.6 - 9.45 - 3.91 ≈ -48 (s.224-236).
6. Which class is predicted and why? - Does Not Love Troll 2, because its log score (-48) is greater than -124 (s.237-238).
7. Determine the correctness: “A more negative log score means a more likely class.” - False; the larger (less negative) log score wins, since log is monotonic increasing, so the ordering of scores is preserved (s.237). (Logic from the slide comparison; the monotonic explanation is added context.)
8. What are the three terms multiplied in the score for a class? - The initial guess/prior $p(\text{class})$ and one likelihood per feature (popcorn, soda pop, candy) given the class (s.224-227).
9. Why might the classification seem wrong when looking at the raw data? - The person ate far more popcorn than the average No-Love person, so by eye they look like a Loves Troll 2 person (s.239-240); the classifier combines all features and the prior (explanation continues in the next chunk).
10. Does the log change which class wins? - No (monotonic), it only avoids underflow.

Added context

Item 7 and 10: the monotonic-function argument is not stated on these slides; it is standard background.

Common confusions

• Larger log score = better: log scores are negative, so -48 is greater than -124 and wins. Do not pick the “bigger magnitude” number.
• Log score vs probability: -124 and -48 are not probabilities; they are log of unnormalised scores (s.222, s.236).
• Rounded vs precise: -48 on s.236-237 is rounded from -47.66 shown on s.238.
• Why a likelihood is a “tiny number”: it is a density value read from a Gaussian curve at the X (far in the tail), not a probability that sums to 1 (s.225-231).
• Sum vs product: after the log, you add the terms; adding the raw (non-log) likelihoods would be wrong.
• The raw-data intuition (s.239-240) can disagree with the classifier because the classifier weighs all features and priors, not popcorn alone.

Slide coverage

Slide	One-line content	Covered in section
211	Use log() to avoid underflow; natural log note	1
212	Take the log of everything (0.5 x 0.06 x 0.004 x tiny)	1
213	Log turns multiplication (red boxed x)…	1
214	…into sum of individual logs	1
215	Log (base e) of 0.5 is…	2
216	…-0.69	2
217	Log of 0.06 is -2.8	2
218	Log of 0.004 is -5.5	2
219	Log of tiny number is -115	2
220	Now we just add this up (red box)	2
221	…and we get -124	2
222	log(Loves Troll 2 Score) = -124	2
223	Start score for Not Loving Troll 2	3
224	Initial guess p(No Love Troll 2) = 0.5	3
225	x L(popcorn = 20 given No Love)	3
226	x L(soda pop = 500 given No Love)	3
227	x L(candy = 25 given No Love)	3
228	Plug in numbers: prior 0.5	3
229	Popcorn likelihood = a tiny number	3
230	Soda pop likelihood = 0.000079	3
231	Candy likelihood = 0.02; take log of everything	3
232	Log turns multiplication…	3
233	…into sum of logs	3
234	Now we just do the math (log terms listed)	3
235	-0.69 + -33.6 + -9.45 + -3.91	3
236	…and we get -48	3
237	-48 > -124 comparison	4
238	Classify as Does Not Love Troll 2 (-47.66)	4
239	NOTE: raw data looks like Loves Troll 2	5
240	Ate a lot more popcorn than average No-Love person…	5

Previous: 03 Text Classification - Chunk 07 (s.181-210) | Next: 03 Text Classification - Chunk 09 (s.241-270) | Up: 03 Text Classification - Index