03 Text Classification - Chunk 09 (s.241-270)

Overview

This chunk finishes the Gaussian Naive Bayes “Troll 2” example (the StatQuest-style cartoon slides): the log-likelihood for candy is so different between the two classes that candy alone can decide the classification, and cross validation is suggested to pick which features to keep (s.241-247). The lecture then switches to a “Quick Review on Linear Regression” (s.248-252), shows why linear regression is a poor classifier (unbounded output, sensitive to imbalanced data; s.253-257), and introduces the logistic (sigmoid) function and Concept - Logistic Regression (s.257-263). The rest explains how parameters are learned: loss function, Concept - Cross-Entropy Loss, and Concept - Gradient Descent (s.264-269), ending with the Concept - Decision Boundary of logistic regression, which is linear (s.270).

Detailed notes

1. Wrap-up of the Gaussian Naive Bayes “Troll 2” example (s.241-247)

Setting: people are split into two groups, those who Love Troll 2 (green) and those who Do Not Love Troll 2 (red). For each person there are three features: Popcorn, Soda Pop, Candy. Each feature has one Gaussian curve per class (green for Loves, red for No Love). A new person (the red figure, the black X on each curve) is being classified. See Concept - Gaussian Naive Bayes.

• Soda pop: the new person drinks as much soda pop as the average person who Loves Troll 2 (the X sits near the centre of the green curve, in the tail of the red one) (s.241).
• Candy: the new person ate a lot more candy than the people who Loved Troll 2 (the green candy curve is a narrow peak at low values; the X lies far out on the broad red curve) (s.242).
• The log-likelihoods for candy are “way different” (s.243):
  • $\log (L(\text{candy}=25\mid \text{Loves}))=-115$
  • $\log (L(\text{candy}=25\mid \text{No Love}))=-3.91$
• This difference is what made the model classify the new person as Does Not Love Troll 2 (s.244).
• Therefore candy can have a much larger say in whether someone Loves Troll 2 than popcorn and soda pop (s.245), and “we might only need candy to make classifications” (s.246).
• We can use Cross Validation to help decide which of popcorn, soda pop and/or candy help make the best classifications (s.247). See Concept - Cross Validation.

2. Quick review on linear regression (s.248-251)

• s.248 is a section-divider slide: “Quick Review on Linear Regression”.
• Supervised-learning pipeline (s.249): Training Set -> Learning Algorithm -> hypothesis $h$. $h$ takes an input $x$ (size of house) and outputs an estimated price $y$.
• Representation of $h$ (s.249): $h(x)=w_0+w_{1}x$, “Linear regression with one variable”, also called Univariate Linear Regression. See Concept - Linear Regression.
• Continuous target (s.250): build an auto-grade algorithm; input feature is one scalar, your previous GPA; target value is your score in H6751. The scatter plot (x: Previous GPA, y: Score in H6751) has a red fitted line through the points.
• Discrete target (s.251): we only want to predict whether you pass H6751; input is the same scalar (previous GPA); target is a binary value (1: pass, 0: fail). This is a classification problem. The plot shows points at 0 (FAIL) for low GPA and 1 (PASS) for high GPA, with a straight blue line fitted through them.

3. Classification setup (s.252)

• Binary classification examples: email (spam / not spam), online transaction (fraud / not fraud).
• Labels: $y=-1$ is the negative class (e.g., not spam, not fraud); $y=1$ is the positive class (e.g., spam, fraud).
• Machine learning is to learn a function from data: $f:X\to Y$.

4. Can we use linear regression for classification? (s.253-257)

• s.253: data plotted as GPA (x, 1.0 to 4.8) vs “K6312 Result” (y, 0 or 1). GPA 1.0-2.8 -> 0; GPA 3.0-4.8 -> 1. (Course code label differs from H6751 on other slides; same example.)
• s.254 “After fitting”: scikit-learn lin_regression.coef_ = 0.37593985 and intercept_ = -0.5902255639097744. A red fitted line crosses a green horizontal threshold at 0.5; the line goes below 0 (about -0.2 at GPA 1.0) and above 1 (about 1.2 at GPA 4.8).
• s.255 Output value is continuous: for classification we want the output to be probabilistic, in range (0, 1), but the output of linear regression is unbounded. Fitted model: $y=0.3759x-0.59$.
  • x = 5 -> y = 1.289
  • x = 4.6 -> y = 1.038
  • x = 1.2 -> y = -0.13
  • What we want: the probability scores of the first two cases close to 1 and the last close to 0.
• s.256 Imbalanced data: add 20 students whose GPA are in the range (6, 7) and who pass H6751. The new linear fit is flatter; the red line now crosses the 0.5 threshold at a higher GPA, so a student with GPA around 3 (circled point, label 1) is wrongly classified (predicted below 0.5). (Axis labels on this figure are left over from another example: x labelled “Age”.)
• Takeaway: linear regression is sensitive to the distribution/imbalance of the data and gives out-of-range values, so we need a function that squashes outputs into (0, 1).

5. The logistic function (s.257-258)

• s.257 Logistic Function: $\sigma (t)=\frac{1}{1+e^{-t}}=\frac{e^t}{1+e^t}$. Input $t\in (-\infty ,+\infty )$; output $\sigma (t)$ is a probabilistic score from 0 to 1. The plot is an S-shaped curve, passing through 0.5 at $t=0$, approaching 0 for $t\ll 0$ and 1 for $t\gg 0$ (x-axis from about -5 to 5). Two overlaid curves (solid red, dashed blue) are drawn; the dashed blue curve is not explained on the slide (UNCLEAR (slide 257): meaning of the dashed vs solid curve).
• s.258: Linear regression $f(x)=w\ast x+b$ becomes logistic regression $f(x)=\frac{1}{1+e^{-(w\ast x+b)}}$ by plugging the linear output into the logistic function.

6. Logistic Regression (s.259-263)

• s.259 is a divider slide: “Logistic Regression”.
• s.260: logistic regression uses the logistic function to model a binary target; it models the distribution $p(y=1\mid x)$ given $x$. Exact parametric formulation:
  • $p(y=1\mid x)=\frac{1}{1+e^{-(wx+b)}}=\frac{e^{(wx+b)}}{1+e^{(wx+b)}}$
  • $p(y=0\mid x)=1-\frac{1}{1+e^{-(wx+b)}}=\frac{1}{1+e^{(wx+b)}}$
  • Then check its performance on the H6751 auto-grade system.
• s.261 “After fitting”: log_regression.coef_ = 1.93582432, intercept_ = [-5.61388646]. The red S-curve rises smoothly from near 0 at GPA 1.0 to about 0.97 at GPA 4.8 and crosses the green 0.5 line at about GPA 2.9, separating the 0 points (GPA 1.0-2.8) from the 1 points (3.0-4.8).
• s.262 Output value is probability score: $t=1.93x-5.6$; $p(y=1\mid x)=e^t/(1+e^t)$; $p(y=0\mid x)=1/(1+e^t)$. Output of predict_proba (columns prob(y=0), prob(y=1)):
  • x = 5: [0.01686949, 0.98313051]
  • x = 4.6: [0.03588451, 0.96411549]
  • x = 1.2: [0.96411521, 0.03588479]
• s.263 Imbalanced data revisited: with the same 20 extra students (GPA 6-7), the logistic curve still gives y=1’s probability > 0.5 for the circled GPA-3 point, i.e., it is classified correctly (contrast with linear regression, s.256). The curve flattens near 1 after GPA 5, so the extreme points do not drag the decision threshold.

7. How to learn the parameters (s.264-265)

• s.264: learning = fitting the data. In Python it is one line: LogisticRegression(random_state=0).fit(X, y) (example with load_iris, from sklearn.linear_model). “Actually, it is an optimization problem (in math perspective).”
• s.265 Optimization: fitting the data -> define a loss function reflecting the fitness of different model parameters; optimization is the process of searching for the minimum point. Figure: 3-D bowl-shaped surface $J(w,b)$ over axes $w$ and $b$ with the minimum marked by a red dot at the bottom.

8. Entropy (cross-entropy) loss (s.266-268)

• Per data point (s.266): $Loss(y,\tilde{y})=-[y\log \tilde{y}+(1-y)\log (1-\tilde{y})]$ where $\tilde{y}=p(y=1\mid x)$.
• Exercise (s.266) “Whether the model prediction is good or not?” with answers on s.267:

Case	Loss	Verdict
y=1, predicted prob of y=1 is 0.9	$-\log 0.9=0.1$	Good fitness, low loss
y=0, predicted prob of y=1 is 0.2	$-\log 0.8=0.22$	Good fitness, low loss
y=1, predicted prob of y=1 is 0.2	$-\log 0.2=1.6$	Bad fitness, high loss
y=0, predicted prob of y=1 is 0.9	$-\log 0.1=2.3$	Bad fitness, high loss

• s.268 Optimization is to reduce loss: for $m$ training points, the cost is a function of the model parameters: $J(w,b)=\frac{{\sum }_{i=1}^{m}Loss({\tilde{y}}^i,y^i)}{m}$ with $\tilde{y}=p(y=1\mid x)=\frac{1}{e^{(-wx+b)}+1}$ (as printed; sign convention as on slide, see also s.260) $=\frac{e^{(wx+b)}}{e^{(wx+b)}+1}$. The arrow shows $\tilde{y}$ plugged into the Loss; same bowl figure of $J(w,b)$.

9. Gradient descent (s.269)

• Update rule: ${\mathbf{w}}_{n+1}={\mathbf{w}}_n-\alpha \nabla J(\mathbf{w})$.
  • ${\mathbf{w}}_{n+1}$ = new parameters guess; ${\mathbf{w}}_n$ = current parameters guess; $\alpha$ = learning rate.
  • $\nabla J(\mathbf{w})$ = gradient of the loss function for $X_n$, “computed by BP algorithm” (backpropagation).
• Figure: 3-D error surface (z: Error 0-0.4, axes Bias and Weight, -2 to 2), coloured red (high) to blue (low) with a white dot in the valley; caption “Like hiking down a mountain”.

10. Decision boundary of logistic regression (s.270)

• Decision by comparing probabilities: $p(y=1\mid \mathbf{x})>p(y=-1\mid \mathbf{x})\iff \frac{p(y=1\mid \mathbf{x})}{p(y=-1\mid \mathbf{x})}>1$.
• Take the logarithm: $\ln \frac{p(y=1\mid \mathbf{x})}{p(y=-1\mid \mathbf{x})}={\mathbf{w}}^{\top }\mathbf{x}+b\to {\mathbf{w}}^{\top }\mathbf{x}$ (bias absorbed into $\mathbf{w}$).
• Decision boundary is linear: ${\mathbf{w}}^{\top }\mathbf{x}+b=0$; $y=+1$ if ${\mathbf{w}}^{\top }\mathbf{x}+b>0$, else $-1$. The threshold (0) is tunable.
• Figure: 2-D scatter with red crosses (upper left) and blue circles (centre and a far cluster at bottom right, near (8, -6)), with two straight lines (green and purple) as example linear boundaries.

flowchart LR
    X["Features x (e.g. GPA)"] --> L["Linear score t = w·x + b"]
    L --> S["Sigmoid σ(t) = 1/(1+e^-t)"]
    S --> P["p(y=1|x) in (0,1)"]
    P --> D{"p > 0.5 ?<br/>(threshold tunable)"}
    D -- yes --> Y1["Class 1 (pass / positive)"]
    D -- no --> Y0["Class 0 (fail / negative)"]
    P -.training.-> LOSS["Cross-entropy loss J(w,b)"]
    LOSS -.gradient descent.-> L

flowchart TD
    A["Init w, b"] --> B["Compute p(y=1|x) for all m points"]
    B --> C["Compute J(w,b) = mean cross-entropy"]
    C --> D["Gradient ∇J"]
    D --> E["w_new = w_old - α ∇J"]
    E --> B

Linear regression vs logistic regression (compare)

Aspect	Linear regression	Logistic regression
Model	$f(x)=wx+b$ (s.258)	$f(x)=\frac{1}{1+e^{-(wx+b)}}$ (s.258)
Output range	Unbounded, e.g. 1.289 or -0.13 (s.255)	Probability in (0,1) (s.257, s.262)
Target	Continuous (s.250)	Binary (s.251, s.260)
Imbalanced data	Wrongly classifies GPA-3 point (s.256)	Still correct (s.263)
Decision boundary	n/a here	Linear, ${\mathbf{w}}^{\top }\mathbf{x}+b=0$ (s.270)

Figures & diagrams

• s.241: three stacked Gaussian plots (Popcorn, Soda Pop, Candy), red vs green curves; Soda Pop highlighted with X near the green mean. Takeaway: soda pop matches the Loves group.
• s.242: Candy plot highlighted; green curve narrow at low candy, red curve broad; X at high candy far from green. Takeaway: candy discriminates.
• s.243-246: same candy plot with log-likelihoods -115 vs -3.91; cartoon captions lead to “candy alone may suffice”.
• s.247: same figure; cross-validation caption.
• s.248: blue divider “Quick Review on Linear Regression”.
• s.249: block diagram Training Set -> Learning Algorithm -> h (house size x -> estimated price y) plus scatter with fitted line and $h(x)=w_0+w_{1}x$.
• s.250: scatter of previous GPA vs score in H6751 with red regression line.
• s.251: binary pass/fail points at 0 and 1 with a straight blue regression line (poor fit).
• s.253: GPA vs K6312 result, step-like 0/1 data.
• s.254: same data with red linear fit and green 0.5 line; code output coef 0.3759, intercept -0.5902.
• s.256: imbalanced data with 20 extra points at GPA 6-7; circled wrongly classified point.
• s.257: S-shaped logistic curve, formula, t range and output range.
• s.258: arrow diagram linear regression -> sigmoid -> logistic regression with formulas.
• s.259: blue divider “Logistic Regression”.
• s.261: S-shaped fitted logistic curve on GPA data with 0.5 line; coef 1.9358, intercept -5.6139.
• s.262: predict_proba table for x = 5, 4.6, 1.2.
• s.263: logistic fit on imbalanced data, circled point has prob > 0.5.
• s.264: scikit-learn code snippet with fit(X, y) boxed.
• s.265, s.268: 3-D bowl $J(w,b)$ with minimum marked.
• s.269: 3-D error surface (bias, weight, error), “hiking down a mountain”.
• s.270: 2-D scatter with linear separating lines.

Key terms

• Concept - Gaussian Naive Bayes
• Concept - Cross Validation
• Concept - Linear Regression
• Concept - Logistic Regression
• Concept - Sigmoid Function
• Concept - Cross-Entropy Loss
• Concept - Gradient Descent
• Concept - Decision Boundary

Formulas

• Log-likelihood for candy=25: $\log L(\text{Loves})=-115$, $\log L(\text{No Love})=-3.91$ (s.243)
• Univariate linear regression: $h(x)=w_0+w_{1}x$ (s.249); $f(x)=w\ast x+b$ (s.258)
• Fitted linear model: $y=0.3759x-0.59$ (s.255)
• Logistic function: $\sigma (t)=\frac{1}{1+e^{-t}}=\frac{e^t}{1+e^t}$ (s.257)
• Logistic regression: $p(y=1\mid x)=\frac{1}{1+e^{-(wx+b)}}$; $p(y=0\mid x)=\frac{1}{1+e^{(wx+b)}}$ (s.260)
• Fitted logistic: $t=1.93x-5.6$ (s.262)
• Loss per point: $Loss(y,\tilde{y})=-[y\log \tilde{y}+(1-y)\log (1-\tilde{y})]$ (s.266)
• Cost: $J(w,b)=\frac{1}{m}{\sum }_{i=1}^{m}Loss({\tilde{y}}^i,y^i)$ (s.268)
• Gradient descent: ${\mathbf{w}}_{n+1}={\mathbf{w}}_n-\alpha \nabla J(\mathbf{w})$ (s.269)
• Decision rule: $\ln \frac{p(y=1\mid \mathbf{x})}{p(y=-1\mid \mathbf{x})}={\mathbf{w}}^{\top }\mathbf{x}+b$; $y=+1$ if ${\mathbf{w}}^{\top }\mathbf{x}+b>0$ (s.270)

Likely exam questions

Style follows Past Exam - Question Bank & Style Guide (short “discuss / describe / explain / determine the correctness” answers).

1. Why can the log-likelihood of one feature dominate a Gaussian Naive Bayes classification? Because the class-conditional likelihoods are multiplied (logs added); a feature such as candy with $\log L=-115$ vs $-3.91$ differs by far more than popcorn or soda pop, so it largely decides the class (s.243-246).
2. How can we decide which features to use (popcorn, soda pop, candy)? Use cross validation to compare classifiers built with different feature subsets (s.247).
3. Discuss why linear regression is not suitable for classification, as discussed in class. Output is unbounded (e.g. y=1.289 at x=5, y=-0.13 at x=1.2) so it is not a probability in (0,1) (s.255); and it is sensitive to imbalanced data: adding 20 students with GPA 6-7 tilts the line so that a GPA-3 student who passes is misclassified (s.256).
4. Write the logistic function and describe its properties. $\sigma (t)=1/(1+e^{-t})$; input in $(-\infty ,\infty )$, output in (0,1), S-shaped, 0.5 at $t=0$ (s.257).
5. Write $p(y=1\mid x)$ and $p(y=0\mid x)$ for logistic regression. See s.260; they sum to 1.
6. Determine the correctness: “Logistic regression is a regression model that outputs unbounded values.” False: it outputs a probability in (0,1) by applying the sigmoid to $wx+b$ (s.258, s.262).
7. Compute the cross-entropy loss for y=1 with predicted prob 0.2. $-\log 0.2=1.6$, high loss / bad fitness; for y=1, prob 0.9: $-\log 0.9=0.1$ (s.267).
8. Describe the steps of gradient descent. Initialize parameters; compute the gradient of $J$; update ${\mathbf{w}}_{n+1}={\mathbf{w}}_n-\alpha \nabla J$; repeat until the minimum (s.265, s.268-269).
9. What is the role of the learning rate $\alpha$? It scales the step taken against the gradient (s.269).
10. Explain why the decision boundary of logistic regression is linear. The log-odds equal ${\mathbf{w}}^{\top }\mathbf{x}+b$; the boundary is where this equals 0 (equal probabilities), a hyperplane; the threshold is tunable (s.270).

Common confusions

• “Logistic regression” is a classifier, not a regression for continuous values; the “regression” part is the linear score $wx+b$ passed through a sigmoid.
• The 0.5 threshold on probability corresponds to ${\mathbf{w}}^{\top }\mathbf{x}+b=0$ on the score; both refer to the same boundary, and the threshold is tunable (s.270).
• Label conventions differ: slides use $y\in \{0,1\}$ for the probability/loss derivation (s.260, s.266) but $y\in \{-1,+1\}$ for the decision-boundary and general classification slides (s.252, s.270).
• Loss vs cost: Loss is per data point (s.266); $J(w,b)$ is the average over $m$ points (s.268).
• Gradient descent subtracts the gradient (moves downhill).
• Logistic fit is less hurt by imbalance than the linear fit (s.256 vs s.263), but this does not mean imbalance is irrelevant.
• Slides mix course codes (H6751 vs K6312) for the same GPA example; they are the same example.

Slide coverage

Slide	Content	Covered in section
241	Naive Bayes: soda pop same as Loves group	Detailed notes 1
242	Candy much higher than Loves group	Detailed notes 1
243	Log-likelihoods candy -115 vs -3.91	Detailed notes 1
244	Difference -> classified Does Not Love	Detailed notes 1
245	Candy has larger say than popcorn/soda	Detailed notes 1
246	Might only need candy	Detailed notes 1
247	Use cross validation to pick features	Detailed notes 1
248	Divider: Quick Review on Linear Regression	Detailed notes 2
249	Pipeline diagram, $h(x)=w_0+w_{1}x$	Detailed notes 2
250	Continuous target (GPA -> score)	Detailed notes 2
251	Discrete target, classification problem	Detailed notes 2
252	Binary classification, $f:X\to Y$	Detailed notes 3
253	Linear regression for classification? data plot	Detailed notes 4
254	After fitting: coef/intercept, line vs 0.5	Detailed notes 4
255	Output continuous/unbounded, examples	Detailed notes 4
256	Imbalanced data, wrongly classified	Detailed notes 4
257	Logistic function	Detailed notes 5
258	Linear -> logistic regression formulas	Detailed notes 5
259	Divider: Logistic Regression	Detailed notes 6
260	Logistic regression definition, $p(y\mid x)$	Detailed notes 6
261	After fitting logistic regression	Detailed notes 6
262	Output is prob. score, predict_proba	Detailed notes 6
263	Imbalanced data with logistic regression	Detailed notes 6
264	How to learn parameters, sklearn fit	Detailed notes 7
265	Optimization, loss surface J(w,b)	Detailed notes 7
266	Entropy loss formula + exercise	Detailed notes 8
267	Entropy loss answers	Detailed notes 8
268	Optimization is to reduce loss, J(w,b)	Detailed notes 8
269	Gradient descent algorithm	Detailed notes 9
270	Decision boundary of logistic regression	Detailed notes 10

Previous: 03 Text Classification - Chunk 08 (s.211-240) | Next: 03 Text Classification - Chunk 10 (s.271-300) | Up: 03 Text Classification - Index