Concept - Maximum Likelihood Estimation

Likelihood (s.9): probability of observing the sample under distribution $d$; under independence $\mathrm{Pr}[x_1,\dots ,x_m]={\prod }_{i=1}^{m}p(x_i)$ (Concept - Likelihood).

MLE principle (s.9): select the distribution maximizing the sample probability. $p_{\ast }=\arg {\max }_{p\in \mathcal{P}}{\prod }_{i=1}^{m}p(x_i)\text{(likelihood)}=\arg {\max }_{p\in \mathcal{P}}{\sum }_{i=1}^m\log p(x_i)\text{(log-likelihood)}$

For hypotheses (s.10): given training data $D$, find $h$ maximizing the likelihood of $D$: $h_{ML}=\arg {\max }_{h\in H}P(D|h)$. The slide then asks what if you have ideas about the hypothesis/parameters, which leads to Concept - Maximum A Posteriori.

Gaussian example (s.11): observations 3.23, 1.23, 0.55, 1.23, …; log-likelihood $l(p)=-\frac{1}{2}m\log (2\pi {\sigma }^2)-\sum \frac{(x_i-\mu {)}^2}{2{\sigma }^2}$; setting derivatives to zero gives $\mu =\frac{1}{m}\sum x_i$ and ${\sigma }^2=\frac{1}{m}\sum (x_i-\mu {)}^2$.

In text classification (s.28): word probabilities are learned by MLE = counting frequencies in the class mega-document, $p(w|h_j)=count(w,h_j)/{\sum }_{d}count(d,h_j)$; fixed for unseen words by Concept - Additive Smoothing.

MLE vs MAP

MLE ignores the prior; MAP multiplies by $P(h)$ (s.19).

Related

Concept - Density Estimation, Concept - Argmax, Concept - Maximum A Posteriori, Concept - Naive Bayes

Source: 03 Text Classification - Chunk 01 (s.1-30)