Concept - Softmax Classifier

Multiclass extension of Concept - Logistic Regression. For 4 classes find $w_{1} .. w_{4}$ with $P (C_{1} ∣ x) : .. : P (C_{4} ∣ x) = e^{w_{1}^{T} x} : .. : e^{w_{4}^{T} x}$ and probabilities summing to 1 (s.288).
Softmax function (s.289): $P (y = i ∣ x) = \frac{e ^{w_{i}^{T} x}}{\sum _{j} e ^{w_{j}^{T} x}}$ , $i = 1.. k$ . It amplifies the largest score and normalizes (bar chart, s.289).
Objective: cross entropy (s.290): $L (w) = - \sum_{i = 1}^{N} \sum_{k = 1}^{K} t_{ik} lo g p (y = k ∣ x)$ , $t_{ik} = 1$ if $y_{i} = k$ else 0 (one-hot targets). Example: true digit 6 one-hot vs computed probabilities with 0.9 at index 6.
Architecture (s.291): inputs + bias -> $z_{j} = w_{j}^{T} x$ for $K$ classes ( $z = W x$ ) -> softmax -> class probabilities.
Contrast with Concept - One-vs-All Classification (independent binary models).

Khaled Notes