Concept - Information Gain

Information Gain (Mutual Information)

$I(X,Y)=E(X)-E(X|Y)=E(Y)-E(Y|X)$

Intuition (s.311): how much knowing Y reduces uncertainty about X, and vice versa. In Concept - ID3 it is the statistical property measuring how well an attribute separates the training examples according to their target classification (s.310); the attribute with the highest gain is tested at the node.

Example - Play Tennis (s.312-314)

• $E(S)=0.940$ for $S=[9+,5-]$ (Concept - Entropy).
• Humidity: $I=0.940-0.5\cdot 0.985-0.5\cdot 0.592=0.151$.
• Outlook: $I=0.940-0.357\cdot 0.971\cdot 2=0.247$ → Outlook wins, becomes root.
• Then recurse on the sunny branch $[2+,3-]$ (s.315).

Weakness

Biased toward attributes with many values (e.g. Day) → fixed by Concept - Gain Ratio (s.321). For real-valued attributes see Concept - Splitting Continuous Attributes.

Related

Concept - Entropy | Concept - Gain Ratio | Concept - Gini Impurity | Concept - Decision Tree

Appears in

• 03 Text Classification - Chunk 11 (s.301-330)