Concept - Prior Probability

Definition

The initial guess of how likely a class is, made before looking at the feature values: “The initial guesses are called Prior Probabilities.” (s.194)

Details

• Can be any probability we want, but a common choice is estimated from the training data (s.188).
• Example: 8 of 16 training people loved Troll 2, so $p(\text{Loves Troll 2})=\frac{8}{8+8}=0.5$ (s.189); likewise $p(\text{No Love Troll 2})=0.5$ (s.192).
• It is the first factor in the class score (s.196).

Related

Concept - Gaussian Naive Bayes, Concept - Likelihood, Concept - Naive Bayes

Source chunk: 03 Text Classification - Chunk 07 (s.181-210)

Spam example (s.67-71, s.81-83)

• Source: 03 Text Classification - Chunk 03 (s.61-90).
• “We start with an initial guess about the probability that any message, regardless of what it says, is a normal message”: $p(N)$ (s.67); can be any value, common choice estimated from training data (s.68).
• 8 of the 12 training messages are normal: $p(N)=\frac{8}{8+4}=0.67$ (s.69). Terminology Alert: this initial guess is called a Prior Probability (s.71).
• Spam: 4 of 12 messages, $p(S)=\frac{4}{4+8}=0.33$ (s.83).
• Class score = prior x word likelihoods, e.g. $0.67\times 0.47\times 0.29=0.09$ (s.77); related: Concept - Posterior Probability.

Priors in the log score (s.224, s.215)

Initial guess $p(\text{No Love Troll 2})=0.5$ (s.224) and $p(\text{Loves Troll 2})=0.5$ (s.211); in the log score the prior enters as $\log (0.5)=-0.69$ (s.216, s.235). See 03 Text Classification - Chunk 08 (s.211-240).

In the “Dear Friend” / “Lunch Money” examples (03 Text Classification - Chunk 04 (s.91-120))

• Priors $p(N)=0.67$ and $p(S)=0.33$ are the starting “guess” that is multiplied by the word likelihoods (s.103). When a likelihood is 0 the prior no longer matters: “it doesn’t matter what value we picked for the initial guess” (s.112-115).

From chunk 02 (s.31)

If every word of a test document is OOV and OOV words are ignored, Naive Bayes reverts to the class priors (Concept - Out of Vocabulary (OOV), s.31).

From chunk 01 (s.1-30)

$P(h)$ is the prior probability of hypothesis $h$ and $P(D)$ the prior probability of the evidence (s.13); the prior “encodes the knowledge/preference” in Concept - Maximum A Posteriori (s.18). Class priors in the text example are document-count ratios: $p(like)=300/1000=0.3$, $p(dislike)=0.7$ (s.27). Evolution from flat prior to peaked posterior: s.15-17 (Concept - Bayes Rule).

Source chunk: 03 Text Classification - Chunk 01 (s.1-30)