Concept - Entropy (TWM)

Entropy (Text & Web Mining)

Measure of the information / impurity in a set of examples (02 Pre-processing s.78, s.80):

$Entropy=-{\sum }_{i=1}^K{p}_i{\log }_2{p}_i$

$K$ = number of classes (“possible actions”), $p_i$ = proportion of class $i$. High entropy = more information (impure, unpredictable); low entropy = less information (pure, predictable).

• Maximum ${\log }_{2}K$ if all $p_i=\frac{1}{K}$ (heterogeneous set); minimum $0$ if one $p_i=1$ (pure) (s.79).
• Binary: $Entropy(S)=-p_{+}{\log }_2{p}_{+}-p_{-}{\log }_2{p}_{-}$; curve is 0 at $p_{+}=0$, 1 at $p_{+}=\frac{1}{2}$, 0 at $p_{+}=1$ (s.80). $Entropy([9+,5-])=0.94$.
• Impurity of splits example (s.77): parent 20P/20N; $f_1$ gives 20P | 20N (pure children), $f_2$ gives 10P,10N | 10P,10N (no help), $f_3$ gives 17P,1N | 3P,19N.
• Intuition (s.81): uncertainty of X, information contained in X, expected bits required to represent X.

Same idea as in the Machine Learning course: ML: Entropy.

Related

Concept - Information Gain (TWM) | Concept - Feature Selection

Appears in

• 02 Pre-processing - Chunk 03 (s.61-90)