02 Pre-processing - Chunk 04 (s.91-120)

Overview

This part of the lecture introduces Concept - Cosine Similarity as a way to measure how similar texts are. It opens with tweets about the movie Troll 2: with a handful of sentences similarity is obvious by eye, but with a month of Twitter traffic a computer is needed (s.91-95). The metric is then explained step by step on a tiny example, “Hello, World!” vs “Hello!”: build a word-count table, plot each phrase as a point, draw lines from the origin, measure the angle ($45^{\circ }$) and take its cosine ($0.71$) (s.97-113). Finally it shows that repeating a word (“Hello Hello Hello!”) leaves the angle, and therefore the similarity, unchanged (s.114-120).

Detailed notes

1. Motivation: similar vs different sentences (s.91-96)

• Four sentences: “I love Troll 2!”, “I like Troll 2.”, “I love Troll 2! I love it!!!” (boxed together) and “Troll 2 is bad.” (s.91).
• By eye the first three are similar - all express a positive feeling toward the movie; the last is from someone who does not like Troll 2 (s.91).
• With few sentences it is easy to see which are similar and which different (s.92).
• Scaling problem: a longer list of tweets (e.g. “Troll 2 is quality entertainment.”, “I really love Troll 2!!!”, “My mom loves Troll 2!!”, “Troll 2 is great!”, “Troll 2 is so-so!!!”, “Troll 2 is awesome!”, “Best Movie Ever: Troll 2!”, “I’m not so sure about Troll 2.”, “Troll 2 = Ugh.” plus the earlier ones) (s.93). Question: what if we collected all of the Twitter traffic for the last month - how do we determine similarities and differences in the tweets? (s.93)
• We can no longer rely on doing things by eye; a computer has to do it (s.94).
• Cosine Similarity “comes in handy” (s.95): a relatively easy to calculate metric that tells how similar or different things are (s.96).

2. Building the count table (s.97-105)

• Start with a “super simple example” (s.97): the phrases “Hello, World!” and “Hello!”; question: how similar is “Hello, World!” to “Hello!”? (s.98)
• Step 1: make a table of the words that appear in the phrases - columns Hello and World (s.99). (This is a Concept - Bag of Words style count table.)
• “Hello, World!”: Hello once (s.100), World once (s.101) -> row (1, 1).
• “Hello!”: Hello one time (s.102), World not at all -> 0 (s.103) -> row (1, 0).

3. Geometric view: points, lines, angle (s.104-111)

• Given the table of counts per word per phrase (s.104), create a 2-dimensional graph: x-axis = number of times Hello is seen (s.105), y-axis = number of times World is seen (s.106).
• “Hello, World!” (1,1) has one of each word, so its point is in the center of the graph (s.107).
• “Hello!” (1,0) lies on the x-axis at 1 (s.108).
• Draw lines from the origin $(0,0)$ to the points (s.109); there is a $45^{\circ }$ angle between the two lines (s.110).
• $\cos (45^{\circ })=0.71$ (s.111), and this is the Cosine Similarity between “Hello, World!” and “Hello!” (s.112).

4. Repeating a word does not change it (s.113-120)

• NOTE: the second phrase was “Hello!” (s.113); what if it were “Hello Hello Hello!”? (s.114)
• The table now has 3 for Hello (it occurs 3 times) (s.115) but still 0 for World (s.116).
• The point for “Hello Hello Hello!” is further out on the x-axis, at (3,0) (s.117).
• However the angle between the two lines is still $45^{\circ }$ (s.118), so the Cosine Similarity is exactly the same as before: $\cos (45^{\circ })=0.71$ (s.119).
• Conclusion: Cosine Similarity is determined entirely by the angle between the lines (s.120) - not by the length of the vectors / amount of repetition.

flowchart TD
    A[Two phrases] --> B[Table of word counts]
    B --> C[Plot each phrase as a point<br/>x = count of word 1, y = count of word 2]
    C --> D[Draw lines from origin 0,0 to each point]
    D --> E[Measure angle between lines]
    E --> F["Cosine Similarity = cos(angle)"]
    F --> G["Hello,World! vs Hello! : 45 deg -> 0.71"]

Figures & diagrams

• s.91 Four Troll 2 sentences; red dotted box around the three positive ones, pink arrows labelling “similar” group and the “bad” sentence as different. Takeaway: similarity is visible by eye on small data.
• s.92 Same sentences without arrows; text only.
• s.93-96 Full list of 13 Troll 2 tweets (positive, neutral “so-so”, negative “bad”, “Ugh”); shows why manual comparison does not scale.
• s.97-98 “Hello, World!” / “Hello!” phrases only.
• s.99-103 Table with header Hello | World; row 1 (pink) = “Hello, World!”, row 2 (green) = “Hello!”. Filled progressively: 1,1 / 1,0.
• s.104-106 Empty 2-D axes: x = Hello (ticks 1, 2), y = World (ticks 1, 2).
• s.107-108 Pink dot at (1,1); green dot at (1,0) on the x-axis.
• s.109-110 Blue lines from origin to both dots; arc labelled ”= $45^{\circ }$”.
• s.111-112 Equation $\cos (45^{\circ })=0.71=$ Cosine Similarity.
• s.113-117 Table row 2 changed to “Hello Hello Hello!” = (3,0); green dot moves to x=3 (axis now to 3).
• s.118-120 Angle still $45^{\circ }$; same cosine 0.71; “determined entirely by the angle”.

Key terms

• Concept - Cosine Similarity
• Concept - Bag of Words (word-count table)
• Concept - Sentiment Analysis (Troll 2 tweets: positive vs negative)

Formulas

• $\text{Cosine Similarity}=\cos (\theta )$, where $\theta$ is the angle between the lines from the origin to the two points (s.111-112, 120).
• Example: $\cos (45^{\circ })=0.71$ for (1,1) vs (1,0) and for (1,1) vs (3,0) (s.111, 119).

Added context

Equivalent vector form (not on slides): $\cos \theta =\frac{A\cdot B}{\Vert A\Vert \Vert B\Vert }$; e.g. $A=(1,1),B=(3,0)$: $3/(\sqrt{2}\cdot 3)=0.71$.

Likely exam questions

1. Why do we need Cosine Similarity when analysing tweets? With a few sentences similarity can be judged by eye, but for e.g. a month of Twitter traffic a computer must do it; Cosine Similarity is an easy-to-compute metric of similarity (s.93-96).
2. Describe, as discussed in class, how to compute Cosine Similarity between two phrases. Build a word-count table; plot each phrase as a point (one axis per word); draw lines from the origin; measure the angle; take its cosine (s.99-112).
3. Determine the Cosine Similarity of “Hello, World!” and “Hello!”. Vectors (1,1) and (1,0), angle $45^{\circ }$, $\cos (45^{\circ })=0.71$ (s.107-112).
4. Determine the Cosine Similarity of “Hello, World!” and “Hello Hello Hello!”. (1,1) vs (3,0): angle still $45^{\circ }$ -> 0.71 (s.117-119).
5. Determine the correctness: “Repeating a word more often makes two phrases more similar/different under Cosine Similarity.” False - only the angle matters, not the counts’ magnitude (s.118-120).
6. What does the cosine similarity depend on? Entirely on the angle between the lines from the origin (s.120).
7. What are the axes of the plot? Each axis is the number of occurrences of one word in the phrase (x = Hello, y = World) (s.105-106).
8. Explain the Troll 2 example. Three positive sentences are similar, “Troll 2 is bad.” is different (s.91); illustrates similarity of opinions in tweets.
9. Which vector representation underlies the table? A word-count (bag-of-words) table (s.99-105).

Common confusions

• Cosine Similarity is about the angle, not the distance between the points: (1,1) and (3,0) are far apart yet give the same value as (1,1) and (1,0) (s.114-120).
• $\cos (45^{\circ })=0.71$ is the similarity itself, not the angle (s.111-112).
• A word absent from a phrase gets count 0 (not omitted) in the table (s.103, 116).
• The two-word, 2-D plot is only a toy example; real texts have many words (dimensions) - UNCLEAR whether the lecture continues this in later slides (not in s.91-120).

Slide coverage

Slide	Content	Covered in
91	Four Troll 2 sentences; first three similar (positive), last different	Detailed notes 1; Figures
92	Few sentences: easy to see similar vs different	1
93	Long tweet list; question about a month of Twitter traffic	1
94	Can’t do it by eye; computer needed	1
95	Cosine Similarity comes in handy	1
96	Cosine Similarity = easy-to-calculate similarity metric	1
97	Start with a super simple example: “Hello, World!” / “Hello!“	2
98	How similar is “Hello, World!” to “Hello!“	2
99	Make a table for words Hello, World	2
100	”Hello, World!”: Hello = 1	2
101	”Hello, World!”: World = 1	2
102	”Hello!”: Hello = 1	2
103	”Hello!”: World = 0	2
104	Given the count table…	3
105	…2-D graph, Hello on x-axis	3
106	World on y-axis	3
107	”Hello, World!” plotted at (1,1)	3
108	”Hello!” plotted at (1,0)	3
109	Lines from origin (0,0) to points	3
110	$45^{\circ }$ angle between lines	3
111	cos(45°) = 0.71	3
112	cos(45°)=0.71 = Cosine Similarity of the phrases	3
113	NOTE: second phrase currently “Hello!“	4
114	What if “Hello Hello Hello!“	4
115	Table has 3 for Hello	4
116	Still 0 for World	4
117	Point further out on x-axis (3,0)	4
118	Angle still $45^{\circ }$	4
119	Cosine Similarity same: 0.71	4
120	Determined entirely by the angle	4

Previous: 02 Pre-processing - Chunk 03 (s.61-90) | Next: 02 Pre-processing - Chunk 05 (s.121-150) | Up: 02 Pre-processing - Index