Concept - Multinomial Naive Bayes

Definition

The Naive Bayes classifier for multinomial models, “suitable for classification with discrete features (e.g., word counts for text classification)”. The multinomial distribution normally requires integer feature counts, but in practice fractional counts such as tf-idf may also work (s.32, scikit-learn documentation screenshot).

scikit-learn (s.32)

sklearn.naive_bayes.MultinomialNB(alpha=1.0, fit_prior=True, class_prior=None)

Added context

alpha is the additive (Laplace/Lidstone) smoothing parameter, see Concept - Additive Smoothing; fit_prior=True learns class priors from the data (Concept - Prior Probability). Not explained on the slide.

Intuition from the lecture example (s.36-60)

Word probabilities per class are obtained from word-count histograms: $p(w\mid c)=\text{count}(w,c)/\text{total words in }c$ (Concept - Word Likelihood).

Related

Concept - Naive Bayes, Concept - Gaussian Naive Bayes (numeric features), Concept - Bag of Words, Concept - TF-IDF

Source chunk: 03 Text Classification - Chunk 02 (s.31-60)

Worked example: “cheap project meds” (s.151-155, from 03 Text Classification - Chunk 06 (s.151-180))

• Training: spam = “buy cheap meds”, “cheap meds online” (6 words); ham = “meeting schedule today”, “project schedule update” (6 words); priors 0.5/0.5; $|V|=9$ (s.152-153).
• Likelihood with Laplace ($\alpha =1$): $P(w\mid c)=\frac{count(w,c)+\alpha }{\text{total words in class}+\alpha |V|}$, denominator $6+9=15$ (s.154).
• Spam score $=0.5\cdot \frac{3}{15}\cdot \frac{3}{15}\cdot \frac{1}{15}\approx 0.00133$; ham score $=0.5\cdot \frac{1}{15}\cdot \frac{1}{15}\cdot \frac{2}{15}\approx 0.000296$ -> spam, about 4.5x more likely (s.155, 160). Normalized: 0.818 vs 0.182 (s.161).
• Repeated words would count repeatedly (s.151).