Concept - Cross-Entropy Loss

(Called “Entropy Loss” on the slides, s.266.)

Per data point: $Loss(y,\tilde{y})=-[y\log \tilde{y}+(1-y)\log (1-\tilde{y})]$, with $\tilde{y}=p(y=1\mid x)$.

Cost over $m$ points (s.268): $J(w,b)=\frac{1}{m}{\sum }_{i=1}^{m}Loss({\tilde{y}}^i,y^i)$.

Worked cases (s.267)

Case	Loss	Verdict
y=1, prob(y=1)=0.9	$-\log 0.9=0.1$	good fit, low loss
y=0, prob(y=1)=0.2	$-\log 0.8=0.22$	good fit, low loss
y=1, prob(y=1)=0.2	$-\log 0.2=1.6$	bad fit, high loss
y=0, prob(y=1)=0.9	$-\log 0.1=2.3$	bad fit, high loss

Intuition

Confident correct predictions cost little; confident wrong ones cost a lot. Optimization = reduce this loss (Concept - Gradient Descent).

Related

Concept - Logistic Regression, Concept - Gradient Descent, Concept - Likelihood

Sources: 03 Text Classification - Chunk 09 (s.241-270)