02 Pre-processing - Chunk 05 (s.121-150)

Overview

This part of the lecture explains Concept - Cosine Similarity, a measure of how similar two texts are, using the tiny vocabulary “Hello” / “World” (slides in the style of StatQuest; a “SQ” logo is in the corner of every slide). Each phrase is turned into a word-count vector (Concept - Bag of Words) and plotted; the angle between the two vectors (not their lengths) gives the similarity as $\cos (\text{angle})$ (s.121). Three cases are shown: identical phrases (cos = 1), no words in common (cos = 0) and partial overlap (cos = 0.71) (s.122-134). The chunk then shows the tedious 4-step geometric procedure (s.135-140) and starts decoding the compact formula with sigmas, index $i$, $n$, $A$ and $B$ (s.141-150); the numeric plug-in is only begun at s.150 and continues in the next chunk. All 30 slides are image-only (no extractable text); every slide was viewed.

Detailed notes

1. Cosine similarity depends on the angle, not the length (s.121)

• Word-count table: “Hello, World!” -> Hello 1, World 1; “Hello Hello Hello!” -> Hello 3, World 0 (s.121).
• Plotted with axes Hello (x) and World (y): pink point at (1,1), green point at (3,0).
• The angle between the two lines from the origin is $45^{\circ }$, $\cos (45^{\circ })=0.71$ = Cosine Similarity.
• Callout: similarity is determined by the angles between the lines “and not by the lengths of the lines” (s.121). So repeating a word more times (longer vector) does not change the similarity.

2. Case 1: identical phrases -> similarity 1 (s.122-127, s.133)

• Both phrases “Hello, World!” -> table rows (1,1) and (1,1) (s.122).
• The two dots are plotted on top of each other (pink + green = “swampy green color” dot at (1,1)) (s.123).
• The angle between the lines from the origin (0,0) to the points is $0^{\circ }$ (s.124) and $\cos (0^{\circ })=1$ = Cosine Similarity (s.125).
• Summary: when both phrases are exactly the same the angle is $0^{\circ }$ and the cosine similarity is 1 (s.126-127); repeated in the summary on s.133.

3. Case 2: nothing in common -> similarity 0 (s.128-132)

• Phrase 1 = “Hello” -> (1,0); phrase 2 = “World” -> (0,1) (s.128).
• Plot: pink dot on the Hello axis at 1, green dot on the World axis at 1 (s.129).
• The angle between them is $90^{\circ }$ (s.130), so $\cos (90^{\circ })=0$ = Cosine Similarity (s.131).
• Summary: when two phrases have absolutely nothing in common the cosine similarity is 0 (s.132).

4. Case 3: partial overlap -> between 0 and 1 (s.134)

• “Hello, World!” (1,1) vs “Hello!” (1,0): some overlap but not exactly the same.
• Angle $45^{\circ }$, $\cos (45^{\circ })=0.71$; the slide states the cosine similarity is between 0 and 1 when phrases partly overlap (s.134).

Case	Phrase A	Phrase B	Angle	Cosine similarity	Slide
Identical	Hello, World! (1,1)	Hello, World! (1,1)	$0^{\circ }$	1	s.122-127, s.133
Partial overlap	Hello, World! (1,1)	Hello! (1,0)	$45^{\circ }$	0.71	s.134
Nothing in common	Hello (1,0)	World (0,1)	$90^{\circ }$	0	s.128-132

5. The manual (geometric) procedure (s.135-140)

Using “Hello, World!” vs “Hello!” (s.135):

1. Step 1) Make a table of the word counts (s.136).
2. Step 2) Plot the points (s.137).
3. Step 3) Figure out the angle ($45^{\circ }$) (s.138).
4. Step 4) Calculate the cosine of the angle: $\cos (45^{\circ })=0.71$ (s.139).

Verdict: this way of computing it “is pretty tedious” (s.140).

6. The formula (s.141-150)

• “The good news”: a relatively simple formula can be computed super quickly on a computer, directly from the table of word counts (s.141).
• Formula (s.142):

$\text{Cosine Similarity}=\frac{{\sum }_{i=1}^n{A}_i{B}_i}{\sqrt{{\sum }_{i=1}^n{A}_i^2}\sqrt{{\sum }_{i=1}^n{B}_i^2}}$

• Notation, introduced one piece at a time:
  • Sigma ($\Sigma$) is shorthand for adding up a bunch of stuff (s.143); the three sigmas are in the numerator and under each square root.
  • $i$ is short for index, used to keep track of the word we are working on (s.144); it starts at 1, the first word in the table, here “Hello” (s.145).
  • $n$ is the number of different words in the phrases; here $n=2$ (Hello and World) (s.146).
  • $A$ and $B$ refer to the two phrases (s.147): $A$ = “Hello, World!” (s.148), $B$ = “Hello!” (s.149).
• Start of the plug-in, $i=1$ (word Hello; the World column is greyed out): numerator term $(1\times 1)+\dots$, denominator $\sqrt{1^2+\dots }\sqrt{1^2+\dots }$ (s.150). The slide is cut here; the World term continues in the next chunk.

Added context

Finishing the calculation for $A=(1,1)$, $B=(1,0)$: numerator $=1\cdot 1+1\cdot 0=1$; denominator $=\sqrt{1^2+1^2}\sqrt{1^2+0^2}=\sqrt{2}$; result $1/\sqrt{2}\approx 0.71$, matching the $\cos (45^{\circ })$ on s.139. This completion is not on slides 121-150.

flowchart LR
    A["Phrase A, Phrase B"] --> C["Word-count table (BoW vectors)"]
    C --> D{"How do the vectors relate?"}
    D -->|"same direction, angle 0"| E["cos 0 = 1 (identical)"]
    D -->|"partial overlap, e.g. 45 deg"| F["cos 45 = 0.71"]
    D -->|"orthogonal, angle 90"| G["cos 90 = 0 (nothing in common)"]

Figures & diagrams

• s.121 Table (Hello/World counts) + 2D plot: pink point (1,1), green point (3,0), $45^{\circ }$ arc, $\cos (45^{\circ })=0.71$; arrows highlight that angle, not length, matters.
• s.122 Table with two identical rows (1,1),(1,1) and empty axes (Hello x-axis 0-2, World y-axis 0-2).
• s.123 Both phrases plotted at the same spot -> olive/“swampy green” dot at (1,1) with a blue line from the origin.
• s.124 Same plot, label ”= 0°” for the angle.
• s.125-127, s.133 Plot with $\cos (0^{\circ })=1$ = Cosine Similarity.
• s.128-129 Table Hello (1,0) / World (0,1); plot: pink dot at Hello=1 on x-axis, green dot at World=1 on y-axis, blue L-shaped lines.
• s.130-132 L-shape with ”= 90°”, then $\cos (90^{\circ })=0$ = Cosine Similarity.
• s.134 Table Hello, World! / Hello!; pink (1,1), green (1,0), $45^{\circ }$ arc, $\cos (45^{\circ })=0.71$.
• s.135-139 Same plot built up step by step (empty axes -> points -> angle -> cosine) alongside “Step 1-4” list.
• s.140-141 Plot faded out, text “good news” about a formula.
• s.142-150 Large formula with sigmas; arrows annotate $\Sigma$, $i$, $n$, $A$, $B$; s.150 shows the first substitution $(1\times 1)+$ over $\sqrt{1^2+}\sqrt{1^2+}$ with the Hello column highlighted in the table.

Key terms

• Concept - Cosine Similarity
• Concept - Bag of Words

Formulas

• Cosine similarity (s.142): $\frac{{\sum }_{i=1}^n{A}_i{B}_i}{\sqrt{{\sum }_{i=1}^n{A}_i^2}\sqrt{{\sum }_{i=1}^n{B}_i^2}}$
• Reference values: $\cos (0^{\circ })=1$ (s.125), $\cos (45^{\circ })=0.71$ (s.134), $\cos (90^{\circ })=0$ (s.131).

Likely exam questions

1. What is cosine similarity and what does it depend on? (s.121) A similarity between two word-count vectors given by the cosine of the angle between them; depends on the angle, not the vector lengths.
2. What is the cosine similarity of two identical phrases? Why? (s.124-127) 1; the vectors coincide, angle $0^{\circ }$, $\cos 0^{\circ }=1$.
3. What is it for two phrases with no words in common? (s.128-132) 0; vectors are orthogonal, angle $90^{\circ }$.
4. What range of values appears for phrases with partial overlap? Example? (s.134) Between 0 and 1, e.g. “Hello, World!” vs “Hello!” gives $\cos 45^{\circ }=0.71$.
5. Describe the four steps of the manual computation. (s.136-139) Word-count table; plot points; figure out the angle; take the cosine.
6. Why use the formula instead of the manual procedure? (s.140-141) Manual is tedious; the formula is quick for a computer and works directly from the word-count table.
7. Write the cosine similarity formula and explain $\Sigma$, $i$, $n$, $A$, $B$. (s.142-149) See Formulas; $\Sigma$ = sum, $i$ = word index starting at 1, $n$ = number of different words, $A$,$B$ = the two phrases.
8. Does repeating a word many times change cosine similarity? (s.121) Not through vector length: “Hello Hello Hello!” (3,0) vs “Hello, World!” (1,1) still has the 45° angle; only direction matters.
9. See also Past Exam - Question Bank & Style Guide (check whether similarity measures appeared; not verified for this chunk).

Common confusions

• Angle vs length: the lines’ lengths do not matter, only the angle between them (s.121).
• Angle vs similarity: small angle = high similarity; $0^{\circ }\to 1$, $90^{\circ }\to 0$ (s.125, s.131). The similarity is the cosine, not the angle itself.
• Identical vs overlapping vs disjoint: 1 vs between 0 and 1 vs 0 (s.133, s.134, s.132).
• $n$ is the number of different words (vocabulary size), not the number of words in a phrase (s.146).
• $A$ and $B$ are phrases (their count vectors), $A_i$, $B_i$ are the counts of word $i$. (s.147-150)

Slide coverage

Slide	Content	Covered in
121	Angle not length matters; Hello,World! (1,1) vs Hello Hello Hello! (3,0), cos 45° = 0.71	Detailed notes 1; Figures
122	Identical phrases table (1,1)/(1,1)	Detailed notes 2
123	Both points plotted on top of each other (swampy green)	Detailed notes 2
124	Angle = 0°	Detailed notes 2
125	cos(0°)=1 = cosine similarity	Detailed notes 2
126	Identical -> angle 0°	Detailed notes 2
127	Cosine similarity is 1	Detailed notes 2
128	Hello vs World: no words in common (table)	Detailed notes 3
129	Points plotted on axes	Detailed notes 3
130	Angle = 90°	Detailed notes 3
131	cos(90°)=0	Detailed notes 3
132	Summary: nothing in common -> 0	Detailed notes 3
133	Summary: identical -> 1	Detailed notes 2
134	Overlap: 45°, cos = 0.71, between 0 and 1	Detailed notes 4
135	Intro “the way we’ve been computing the cosine similarity…” (empty plot)	Detailed notes 5
136	Step 1: table of word counts	Detailed notes 5
137	Step 2: plot the points	Detailed notes 5
138	Step 3: figure out the angle	Detailed notes 5
139	Step 4: calculate cosine of the angle	Detailed notes 5
140	”…is pretty tedious”	Detailed notes 5
141	Good news: simple formula from word-count table	Detailed notes 6
142	Cosine similarity formula	Detailed notes 6; Formulas
143	Sigmas = adding up stuff	Detailed notes 6
144	i = index	Detailed notes 6
145	i starts at 1 (Hello)	Detailed notes 6
146	n = number of different words = 2	Detailed notes 6
147	A and B refer to the two phrases	Detailed notes 6
148	A = “Hello, World!”	Detailed notes 6
149	B = “Hello!”	Detailed notes 6
150	i = 1 plug-in: (1x1)+ / sqrt(1²+) sqrt(1²+)	Detailed notes 6

Previous: 02 Pre-processing - Chunk 04 (s.91-120) | Next: 02 Pre-processing - Chunk 06 (s.151-180) | Up: 02 Pre-processing - Index