Concept - Underflow

Definition

“Every computer has a limit to how close a number can get to 0 before it can no longer accurately keep track of that number. When a number gets smaller than that limit, we run into Underflow problems and errors occur.” (s.209-210)

Why it matters

Multiplying a prior by many small likelihoods (e.g. $0.5\times 0.06\times 0.004\times \text{tiny}$) gives extremely small numbers (s.206). Remedy: “take the log() of everything to prevent … Underflow” (s.207).

Added context

Taking logs turns the product into a sum of logs, which stays in a representable range; the slides in this chunk only state the remedy, not the calculation.

Related

Concept - Likelihood, Concept - Gaussian Naive Bayes

Source chunk: 03 Text Classification - Chunk 07 (s.181-210)

The log trick worked through (s.211-222, s.231-236)

• Any log works; the natural log (base $e$) is most common in statistics and machine learning (s.211).
• Take the log of the whole score; the log turns the multiplication into a sum of individual logs (s.212-214): $\log (a\cdot b\cdot c)=\log a+\log b+\log c$.
• Loves Troll 2: $\log (0.5)+\log (0.06)+\log (0.004)+\log (\text{tiny})=-0.69-2.8-5.5-115=-124$ (s.215-221). The tiny number only becomes -115, which is easily representable.
• Does Not Love Troll 2: $-0.69-33.6-9.45-3.91\approx -48$ (-47.66) (s.235-236, s.238).
• Compare log scores: the larger (less negative) wins, so -48 beats -124 (s.237).

Source chunk: 03 Text Classification - Chunk 08 (s.211-240)

Seen in 03 Text Classification - Chunk 04 (s.91-120)

• Even a 5-word message gives a tiny normal score: $0.67\times 0.18\times 0.06^4=0.000002$ (s.106). Longer messages shrink further, which is why implementations use log-probabilities. (Not stated on s.91-120; this link is added context.)