Concept - Gradient Descent

Definition (s.269): iterative optimisation that repeatedly moves the parameters against the gradient of the loss.

${\mathbf{w}}_{n+1}={\mathbf{w}}_n-\alpha \nabla J(\mathbf{w})$

• ${\mathbf{w}}_{n+1}$: new parameters guess; ${\mathbf{w}}_n$: current parameters guess; $\alpha$: learning rate.
• $\nabla J(\mathbf{w})$: gradient of the loss function, “computed by BP algorithm” (backpropagation).

Intuition

“Like hiking down a mountain” (s.269): the loss surface $J(w,b)$ is a bowl (s.265, s.268); each step goes downhill toward the minimum. The slide’s error surface has axes Bias, Weight, Error.

Context

Fitting = define a loss, then optimisation searches for its minimum (s.265). Used to train Concept - Logistic Regression with Concept - Cross-Entropy Loss.

Related

Concept - Cross-Entropy Loss, Concept - Logistic Regression

Sources: 03 Text Classification - Chunk 09 (s.241-270)