Concept - Gaussian Naive Bayes

Definition

“Gaussian Naive Bayes is named after the Gaussian distributions that represent the data in the Training Dataset.” (s.181) Each numeric feature is modelled per class by a Gaussian curve; a new example is scored per class by prior $\times$ likelihoods of its feature values.

Worked example (Troll 2, s.181-210)

• Features: popcorn (g), soda pop (ml), candy (g); classes: Loves / Does not Love Troll 2; 16 training people (8 + 8) (s.181, s.189).
• New person: 20 g popcorn, 500 ml soda pop, 25 g candy (s.183-185).
• $\text{Score(Loves)}=p(\text{Loves})\times L(\text{popcorn}=20\mid \text{Loves})\times L(\text{soda}=500\mid \text{Loves})\times L(\text{candy}=25\mid \text{Loves})=0.5\times 0.06\times 0.004\times (\text{very small})$ (s.195-206).
• Take the log to avoid Concept - Underflow (s.207).

Related

Concept - Naive Bayes, Concept - Prior Probability, Concept - Likelihood, Concept - Underflow

Source chunk: 03 Text Classification - Chunk 07 (s.181-210)

Worked classification result (s.211-240)

Example with features popcorn = 20 g, soda pop = 500 ml, candy = 25 g. Log scores: Loves Troll 2 = -124 (s.222); Does Not Love Troll 2 = -48 / -47.66 (s.236, s.238), using prior 0.5 and likelihoods (soda pop 0.000079, candy 0.02, popcorn “a tiny number”) (s.228-231). Since -48 > -124, classify as Does Not Love Troll 2 (s.237-238). Caveat: by eye the person looks like a Loves person because of high popcorn intake (s.239-240). Uses the log trick, see Concept - Underflow.

Source chunk: 03 Text Classification - Chunk 08 (s.211-240)

Assumptions and building the Gaussians (s.162-180, from 03 Text Classification - Chunk 06 (s.151-180))

• Assumptions (s.162): features are continuous and follow a normal (Gaussian) distribution given the class. Suits sensor readings, continuous measurements, image pixel intensities; could also be applied to continuous embeddings (word2vec, GloVe, BERT) (see Concept - Word Embeddings).
• Troll 2 set-up (s.164-169): predict whether someone loves the 1990 movie Troll 2; features popcorn (g), soda pop (ml), candy (g) per person, in one table per class.
• Parameters per feature per class = mean and standard deviation (s.170-175): popcorn, Loves: mean 24, SD 4; Does not love: mean 4, SD 2. Then draw the Gaussian for soda pop (s.176-178) and candy (s.179-180) for both classes.
• Candy: Loves = narrow curve near low values; Does not love = wide flat curve around high values (s.179-180).

Candy dominates; feature selection (s.241-247, from 03 Text Classification - Chunk 09 (s.241-270))

For the new person (candy = 25): soda pop looks like the Loves group (s.241), but candy is far above the Loves group’s (s.242). $\log L(\text{candy}=25\mid \text{Loves})=-115$ vs $\log L(\text{candy}=25\mid \text{No Love})=-3.91$ (s.243), which is what makes the classifier say Does Not Love Troll 2 (s.244). So candy can have a much larger say than popcorn/soda pop (s.245) and might be all that is needed (s.246). Use Concept - Cross Validation to decide which features to keep (s.247).