02 Pre-processing - Chunk 03 (s.61-90)

Overview

This part of the lecture finishes the WordNet relations (s.61-62) and then moves to feature generation with the Concept - Bag of Words model (s.63-71): documents become rows of a matrix, features become columns, entries can be presence / frequency / thresholded frequency / Concept - TF-IDF values, multiword features and Concept - Sparse Matrix (CSR) storage are introduced, and the pros and cons of BoW are listed. Because BoW suffers from high dimensionality, the lecture introduces Concept - Dictionary Reduction (s.72-74) and focuses on feature selection by attribute ranking using Concept - Information Gain (TWM) based on Concept - Entropy (TWM) (s.75-88), with a fully worked weather (“play tennis”) example. The chunk ends by opening the section on the Concept - Document-Term Matrix and Concept - Cosine Similarity (s.89-90), which continues in the next chunk.

Detailed notes

1. WordNet relations (s.61-62)

• Each WordNet entry is connected with other entries in a graph through relations (s.61). See Concept - WordNet Relations.
• s.61 shows two WordNet web-browser entries (nouns):
  • breakfast = “the first meal of the day (usually in the morning)”. Links shown: direct hyponym / full hyponym; direct hypernym (bold) / inherited hypernym / sister term -> meal, repast (“the food served and eaten at one time”); derivationally related form.
  • course = “part of a meal served at one time”, example “she prepared a three course meal”. Links shown: direct hyponym / full hyponym; direct hypernym / inherited hypernym / sister term; part holonym (bold) -> meal, repast. (The bottom of this screenshot is cut off on the slide.)
• s.62 table: relations in the database of nouns:

Relation	Definition	Example
Hypernym	From concepts to superordinate	breakfast -> meal
Hyponym	From concepts to subtypes	meal -> lunch
Has-Member (member meronym)	From groups to their members	faculty -> professor
Member-Of (member holonym)	From members to their groups	co-pilot -> crew
Has-Part (part meronym)	From wholes to parts	table -> leg
Part-Of (part holonym)	From parts to wholes	course -> meal
Antonym	Opposites	leader -> follower

2. Bag-of-Words: vector representation (s.63-66)

• s.63 is a section title: “Bag-of-Words: counting is everything”.
• Vector representation for documents (s.64):
  • Without any deep analysis of the linguistic content of the documents, each document can be described by features that represent the most frequent tokens.
  • Each row is a document, each column is a feature.
  • A cell in the csv/excel file is a measurement of a feature (column) for a document (row).
• BoW steps (s.65): (1) build the vocabulary, i.e. the set of all words in the corpus; (2) count the occurrences of words in each document.
  • Example corpus: “The cat and the dog play” / “The cat is on the mat”. Vocab: and, the, cat, dog, play, on, mat, is. See the figure section for the count vectors and a slide inconsistency.
• How to define document features (entry values in the matrix) (s.66):
  • Presence (0 or 1)
  • Frequencies (0, 1, 2, 3, …)
  • Thresholding frequencies - three values: 0 (does not exist), 1 (occurred once), 2 (occurred 2 or more times). (The slide spells “exit”; meant “exist”.)

3. TF-IDF (s.67-68)

• $\text{tf-idf}(w)=tf(w)\times idf(w)$, with $idf(w)=\log \left(1+\frac{N}{df(w)}\right)$ (s.67). (The slide writes “Tf-id(w)”.)
  • The tf-idf weight of word $w$ is the term frequency (word count) modified by a scale factor for the importance of the word.
  • The scale factor is the inverse document frequency: it checks the number of documents containing $w$ ($df(w)$) and reverses the scaling.
  • $N$ = number of documents.
• Intuitive logic (s.68):
  • Captures the importance of a word to a document in a corpus.
  • Importance is proportional to the number of times the word appears (in the document).
  • Importance is inversely proportional to the number of documents containing the word.
  • So a word that appears in many documents is unimportant and its scale is lowered, perhaps near zero, e.g. “the”, “I”, “on”, “document”.

4. Multiword features (s.69)

• Besides single words, features can be multiword units; a variety of measures can be used, e.g. frequent n-grams such as “text mining”, “hip hop”.
• Another method: an Association Measure (AM) for a multiword $T$, where $size(T)$ = number of words in phrase $T$ and $freq(T)$ = number of times $T$ occurs in the document collection (formula below; see Concept - Multiword Features).
• Generally multiword features are not found too frequently in a collection, but when they occur they are often highly predictive.

5. Compressed Sparse Row (CSR) representation (s.70)

• When preparing the feature matrix, most entries will be zero: most documents contain only a small subset of the vocabulary’s words.
• Rather than storing all the zeros, represent the matrix as a set of sparse vectors: each row is a list of pairs (column number, nonzero feature value). See Concept - Sparse Matrix (CSR) and the worked example in the figures.

6. Pros and cons of Bag-of-Words (s.71)

Pros	Cons
Simple	Order of words does not matter
Surprisingly effective	Cannot capture syntactic / semantic information
Fast	High dimensionality (highlighted in red)

7. Dictionary reduction (s.72-74)

• s.72 is a section title (“Dictionary Reduction”).
• s.73: Dictionary reduction is also called feature reduction techniques, and is needed due to the curse of high dimensionality (Concept - Curse of Dimensionality). For BoW models, the techniques listed are:
  1. Local dictionary
  2. Removing stopwords (Concept - Stopword Removal)
  3. Frequent words
  4. Feature selection (highlighted in red)
  5. Token reduction (stemming and synonyms) (Concept - Stemming)
  6. Feature transformation (PCA, or topic models)
• s.74 Feature selection by attribute ranking:
  • Select a set of features (e.g. words) to form a local dictionary.
  • Rank feature attributes by their predictive ability for the category under consideration; e.g. Sports: soccer, football, etc.; Travel: airport, cruise, etc.
  • In this approach, simply select the top-ranking features.
  • Feature selection approaches listed: Document Frequency, Information Gain, Mutual Information, CHI (chi-square), and “A survey” (a hyperlink on the slide, target not visible).
  • Only Information Gain is developed in this chunk. See Concept - Feature Selection.

7b. Information Gain based feature selection (s.75-88)

• s.75: section title “Feature Selection based on Information Gain”.
• Widely used example (s.76): the 14-row weather dataset with features/attributes Outlook, Temperature, Humidity, Windy (red box) and label Play (blue box). “You may think the most important feature is the one that can be most related to the label.”
• Impurity of splits (s.77): set $S$ contains 20 positive (P) and 20 negative (N) occurrences. Each datum has three binary features $f_1,f_2,f_3$, giving three possible splits. $S_1$ = feature is 0, $S_2$ = feature is 1.

Feature	$S_1$	$S_2$	Reading
$f_1$	20P	20N	both children pure
$f_2$	10P, 10N	10P, 10N	both children as mixed as the parent
$f_3$	17P, 1N	3P, 19N	children mostly pure

(The “reading” column is added interpretation; the slide only lists the counts and uses them to motivate impurity.)

• Entropy (s.78): measure of the information in a set of examples: $Entropy=-{\sum }_{i=1}^K{p}_i{\log }_2{p}_i$, where $i=\{1,\dots ,K\}$, $K$ = number of possible actions (classes), $p_i$ = proportion of each action $i$ in the example set. Slide example: $Entropy([9\ast ,5+,6-])=-\frac{9}{20}{\log }_2\frac{9}{20}-\frac{5}{20}{\log }_2\frac{5}{20}-\frac{6}{20}{\log }_2\frac{6}{20}$ (value not given). High entropy = more information; low entropy = less information. A Khan Academy video thumbnail “Information entropy” is embedded as a pointer.
• Properties of entropy (s.79):
  • Maximized when events are heterogeneous (impure): a set of many mixed classes (say red/green/blue circles) is unpredictable -> high entropy. $Entropy={\log }_{2}K$ if all $p_i=\frac{1}{K}$.
  • Minimized when events are homogeneous (pure): a set of only one class (say all blue) is extremely predictable -> low entropy. $Entropy=0$ if one $p_i=1$ and the rest are zeros.
• Entropy for the binary case (s.80): $S$ is a sample of training examples, $P_{+}$ / $P_{-}$ = proportion of positive / negative examples. Entropy measures the impurity of $S$: $Entropy(S)=-p_{+}{\log }_2{p}_{+}-p_{-}{\log }_2{p}_{-}$. Example: $Entropy([9+,5-])=-\frac{9}{14}{\log }_2\frac{9}{14}-\frac{5}{14}{\log }_2\frac{5}{14}=0.94$. Plot: see figures.
• Information gain (s.81) - three quantities:
  • Entropy $E(X)=-{\sum }_{i=1}^{K}p(X=X_i){\log }_{2}p(X=X_i)$. Intuition: uncertainty of $X$, information contained in $X$, expected information bits required to represent $X$.
  • Conditional entropy $E(X|Y)={\sum }_{i=1}^{K}p(Y=Y_i)E(X|Y=Y_i)$. Intuition: given $Y$, how much uncertainty remains in $X$.
  • Mutual information (= information gain) $I(X,Y)=E(X)-E(X|Y)=E(Y)-E(Y|X)$. Intuition: how much knowing $Y$ reduces uncertainty about $X$, and vice versa. Red annotation: “High IG, More Entropy Removed”.
• IG worked example - Humidity (s.82-86), on the 14-row weather data with label Play (5 No, 9 Yes):
  • s.82: overall entropy $E=-\sum p_k{\log }_2{p}_k=-\frac{5}{14}{\log }_2\frac{5}{14}-\frac{9}{14}{\log }_2\frac{9}{14}=0.94$ (No count 5, Yes count 9 are coloured in the formula).
  • s.83: split on Humidity (High rows in red, Normal rows in green). Formula as written: $\Delta E(Humidity)=E-\frac{m_{i=H}}{m}E(i=N)-\frac{m_{i=N}}{m}E(i=H)=0.94-\frac{7}{14}{H}_L-\frac{7}{14}{H}_R$. (The slide’s subscripts on the weights/entropies look swapped; the intended meaning is weighted entropy of each child, 7 of 14 rows each. $H_L$ = entropy of Humidity=Normal group, $H_R$ = Humidity=High group, as shown by the numbers on s.84-85.)
  • s.84: Humidity = Normal rows highlighted: 7 rows, 6 Yes and 1 No: $H_L=-\frac{6}{7}{\log }_2\frac{6}{7}-\frac{1}{7}{\log }_2\frac{1}{7}$.
  • s.85: $H_L=0.592$. Humidity = High rows highlighted: 7 rows, 3 Yes and 4 No: $H_R=-\frac{3}{7}{\log }_2\frac{3}{7}-\frac{4}{7}{\log }_2\frac{4}{7}=0.985$.
  • s.86: $0.94-\frac{7}{14}(0.592)-\frac{7}{14}(0.985)=0.94-0.296-0.4925=0.1515$ = information gain of Humidity.
• Exercise on the slide (s.87): (1) compute the information gain for the remaining three features: outlook, temperature, windy; (2) Should we select features with high IG or low IG? (Answer by the s.81 intuition: high IG = more entropy removed, so select high IG; see Likely exam questions.)
• Text-mining version (s.88): the previous features/attributes become words / terms. The information gain of a term is the expected reduction in entropy caused by partitioning the sample documents according to the term:

$IG(t)=-{\sum }_{i=1}^{m}p(c_i)\log p(c_i)+p(t){\sum }_{i=1}^{m}p(c_i|t)\log p(c_i|t)+p(\bar{t}){\sum }_{i=1}^{m}p(c_i|\bar{t})\log p(c_i|\bar{t})$

Symbol	Meaning (s.88)
$t$	a term
$m$	total number of classes
$p(c_i)$	percentage of documents in category $c_i$ from total sample documents
$p(t)$	percentage of documents in which term $t$ is present
$p(\bar{t})$	percentage of documents in which term $t$ is absent
$p(c_i\Vert t)$	conditional probability of category given term $t$
$p(c_i\Vert \bar{t})$	conditional probability of category given term $t$ is absent

8. Start of the document-term matrix and cosine similarity section (s.89-90)

• s.89: section title “Document-term Matrix & Cosine Similarity”.
• s.90: an image-only slide (appears to be a video frame from a StatQuest-style tutorial, “SQ” logo at top left): “Imagine we saw these sentences about the 1990 hit movie, Troll 2”: “I love Troll 2!” / “I like Troll 2.” / “I love Troll 2! I love it!!!” / “Troll 2 is bad.” Two arrows point from the caption to the first two sentences. The takeaway is only the setup: a tiny corpus of short opinion sentences that will be turned into counts/vectors (continues in the next chunk, see Concept - Document-Term Matrix and Concept - Cosine Similarity).

Figures & diagrams

• s.61 Two WordNet browser screenshots (breakfast, course) showing relation links (direct hyponym, direct hypernym -> meal, repast, part holonym, derivationally related form). Takeaway: WordNet is a graph of linked synsets.
• s.62 Table of noun relations (hypernym, hyponym, member/part meronym and holonym, antonym) with examples (see table above).
• s.65 Three boxes: corpus (the two sentences), vocab. (and, the, cat, dog, play, on, mat, is), countVec (two rows of 8 cells). Row 1 on the slide: 1 2 1 1 1 0 0 0 (correct for “The cat and the dog play”). Row 2 on the slide: 1 2 0 0 1 1 1 1.

  Added context

  Row 2 as printed does not match the vocabulary order for “The cat is on the mat” (counts per vocab word would be and=0, the=2, cat=1, dog=0, play=0, on=1, mat=1, is=1 -> 0 2 1 0 0 1 1 1). UNCLEAR (slide 65): the printed row 2 looks like a slide typo; the slide’s sum of the two rows’ the-column (2) is correct, the others are not.

• s.67 idf formula $idf(w)=\log (1+\frac{N}{df(w)})$ rendered next to the tf-idf definition.
• s.69 Association-measure formula (see Formulas).
• s.70 CSR example: left a 3x4 matrix, right its pair-list form:

Row	Dense row	Sparse (column, value) pairs
1	0 15 0 3	(2,15) (4,3)
2	12 0 0 0	(1,12)
3	8 0 5 2	(1,8) (3,5) (4,2)

(Columns are numbered from 1.)

• s.73 Curse-of-dimensionality drawing: 1-D line of 5 cubes (axis X), a 2-D 5x5 grid (X, Y), and a 3-D cube grid (X, Y, Z): the space to cover grows quickly with each added dimension. Below it, a feature selection strip diagram: “All Features” = 7 coloured boxes -> “Feature Selection” = same strip with 4 boxes crossed out (1st, 3rd, 4th, 5th) -> “Final Features” = the 3 remaining boxes (pink, teal, blue).
• s.74 Flow diagram “Approach used on text data”: Vector of features -> each individual feature goes to a Scoring measure (on individual features) -> feature score returns to Sort by score (after all features are scored) -> First k features -> Learn model (ML algorithm).
• s.76 14-row weather table; red box = features (Outlook, Temperature, Humidity, Windy), blue box = label (Play).

Outlook	Temperature	Humidity	Windy	Play
Sunny	Hot	High	False	No
Sunny	Hot	High	True	No
Overcast	Hot	High	False	Yes
Rainy	Mild	High	False	Yes
Rainy	Cool	Normal	False	Yes
Rainy	Cool	Normal	True	No
Overcast	Cool	Normal	True	Yes
Sunny	Mild	High	False	No
Sunny	Cool	Normal	False	Yes
Rainy	Mild	Normal	False	Yes
Sunny	Mild	Normal	True	Yes
Overcast	Mild	High	True	Yes
Overcast	Hot	Normal	False	Yes
Rainy	Mild	High	True	No

• s.78 Khan Academy “Information entropy” video thumbnail (decorative pointer).
• s.80 Binary entropy curve: x-axis $P_{(+)}$ from 0 to 1, y-axis entropy from 0 to 1; inverted-U curve: 0 at $P_{(+)}=0$, maximum 1 at $P_{(+)}=\frac{1}{2}$, 0 at $P_{(+)}=1$. Takeaway: entropy (impurity) is highest for a 50/50 split and zero for a pure set.
• s.82-86 Same weather table, progressively highlighted: s.82 Play column coloured (No red / Yes green) with the overall entropy; s.83 Humidity values coloured red (High) / green (Normal); s.84 Normal rows bolded; s.85 High rows bolded; s.86 final arithmetic.
• s.90 Four short sentences about Troll 2 (see detailed notes).

flowchart LR
    V[Vector of features<br/>words] --> SC[Scoring measure<br/>on each individual feature<br/>e.g. Information Gain]
    SC --> SO[Sort by score]
    SO --> K[Keep first k features<br/>local dictionary]
    K --> M[Learn model<br/>ML algorithm]

flowchart TD
    BOW[Bag-of-Words matrix<br/>rows = documents, cols = words] --> P[Problem: high dimensionality]
    P --> DR[Dictionary reduction / feature reduction]
    DR --> A[Local dictionary]
    DR --> B[Remove stopwords]
    DR --> C[Frequent words]
    DR --> D[Feature selection<br/>rank by DF, IG, MI, CHI]
    DR --> E[Token reduction<br/>stemming, synonyms]
    DR --> F[Feature transformation<br/>PCA, topic models]
    D --> IG[Information Gain<br/>IG = E - weighted child entropies]

flowchart TD
    S["S: 14 days<br/>9 Yes, 5 No<br/>Entropy 0.94"] --> N["Humidity = Normal<br/>7 rows: 6 Yes, 1 No<br/>H = 0.592"]
    S --> H["Humidity = High<br/>7 rows: 3 Yes, 4 No<br/>H = 0.985"]
    N --> R["IG = 0.94 - 7/14 x 0.592 - 7/14 x 0.985 = 0.1515"]
    H --> R

Key terms

• Concept - WordNet Relations
• Concept - Bag of Words
• Concept - TF-IDF
• Concept - Multiword Features
• Concept - Sparse Matrix (CSR)
• Concept - Document-Term Matrix
• Concept - Dictionary Reduction
• Concept - Curse of Dimensionality
• Concept - Feature Selection
• Concept - Entropy (TWM)
• Concept - Information Gain (TWM)
• Concept - Stopword Removal
• Concept - Stemming
• Concept - Cosine Similarity (introduced at the very end, s.89)

Formulas

• tf-idf: $\text{tf-idf}(w)=tf(w)\cdot idf(w)$, $idf(w)=\log \left(1+\frac{N}{df(w)}\right)$ (s.67)
• Association measure for multiword $T$: $AM(T)=\frac{size(T){\log }_{10}(freq(T))freq(T)}{{\sum }_{wor{d}_i\in T}freq(wor{d}_i)}$ (s.69)
• Entropy: $Entropy=-{\sum }_{i=1}^K{p}_i{\log }_2{p}_i$ (s.78); max $={\log }_{2}K$ when all $p_i=\frac{1}{K}$; $=0$ when one $p_i=1$ (s.79)
• Binary entropy: $Entropy(S)=-p_{+}{\log }_2{p}_{+}-p_{-}{\log }_2{p}_{-}$; $Entropy([9+,5-])=0.94$ (s.80)
• Conditional entropy: $E(X|Y)={\sum }_{i}p(Y=Y_i)E(X|Y=Y_i)$ (s.81)
• Mutual information / information gain: $I(X,Y)=E(X)-E(X|Y)=E(Y)-E(Y|X)$ (s.81)
• IG of a split (Humidity example): $0.94-\frac{7}{14}(0.592)-\frac{7}{14}(0.985)=0.1515$ (s.83-86)
• IG of a term $t$: $IG(t)=-{\sum }_{i=1}^{m}p(c_i)\log p(c_i)+p(t){\sum }_{i}p(c_i|t)\log p(c_i|t)+p(\bar{t}){\sum }_{i}p(c_i|\bar{t})\log p(c_i|\bar{t})$ (s.88)

Added context

Answer to the s.87 exercise (not on the slides), computed from the s.76 table with the same method as Humidity: IG(Outlook) = 0.247, IG(Windy) = 0.048, IG(Temperature) = 0.029, plus IG(Humidity) = 0.152 from the slides. Outlook is therefore the best feature and Temperature the weakest. Working: Outlook splits into Sunny (2 Yes/3 No, H=0.971), Overcast (4/0, H=0), Rainy (3/2, H=0.971): $0.94-\frac{5}{14}0.971-\frac{4}{14}0-\frac{5}{14}0.971=0.247$. Windy: False (6 Yes/2 No, H=0.811), True (3/3, H=1): $0.94-\frac{8}{14}0.811-\frac{6}{14}1=0.048$. Temperature: Hot (2/2, H=1), Mild (4/2, H=0.918), Cool (3/1, H=0.811): $0.94-\frac{4}{14}1-\frac{6}{14}0.918-\frac{4}{14}0.811=0.029$. The slide formula on s.88 is printed with the sign convention where the first term is negative and the two conditional terms positive (the $\sum p\log p$ terms are themselves negative), i.e. it equals $H(C)-H(C|T)$. Compare the Machine Learning course treatment in ML: Information Gain.

Likely exam questions

Style reference: Past Exam - Question Bank & Style Guide (feature selection / information gain style questions fit its “discuss / compute / determine” format).

1. Describe the bag-of-words steps and what the matrix looks like. (s.64-65) Build the vocabulary (all words in the corpus), count each word’s occurrences in each document; each row is a document, each column a feature/word, each cell the feature’s value for that document.
2. How can document features (matrix entries) be defined? (s.66) Presence (0/1), frequencies (0,1,2,3…), thresholded frequencies with three values (0 = absent, 1 = once, 2 = two or more times); also tf-idf (s.67).
3. Explain tf-idf and its intuition, as discussed in class. (s.67-68) $tf\times idf$ with $idf=\log (1+N/df)$; importance grows with the number of times a word appears in the document and shrinks with the number of documents containing it; words like “the”, “I”, “on” get a scale near zero.
4. What are the advantages and disadvantages of BoW? (s.71) Pros: simple, surprisingly effective, fast. Cons: word order ignored, no syntactic/semantic information, high dimensionality.
5. Why and how is a sparse (CSR) representation used? (s.70) Most matrix entries are zero because each document uses few vocabulary words; store only (column, nonzero value) pairs per row instead of all zeros. Example: row 8 0 5 2 -> (1,8)(3,5)(4,2).
6. What are multiword features and why are they useful? (s.69) E.g. frequent n-grams like “text mining”; AM(T) scores them. Rarely frequent, but highly predictive when they occur.
7. List dictionary (feature) reduction techniques for BoW. (s.73) Local dictionary, removing stopwords, frequent words, feature selection, token reduction (stemming, synonyms), feature transformation (PCA, topic models). Reason: curse of high dimensionality.
8. Describe feature selection by attribute ranking. (s.74) Score each feature individually with a scoring measure (document frequency, information gain, mutual information, chi-square), sort by score, keep the top k, learn the model on those.
9. Define entropy and describe its extreme values. (s.78-80) $-\sum p_i{\log }_2{p}_i$, a measure of impurity/uncertainty; max ${\log }_{2}K$ for uniform distribution (heterogeneous set), 0 for a pure set; binary entropy is 1 at $p_{+}=0.5$ and 0 at $p_{+}\in \{0,1\}$.
10. Compute $Entropy([9+,5-])$. (s.80) $-\frac{9}{14}{\log }_2\frac{9}{14}-\frac{5}{14}{\log }_2\frac{5}{14}=0.94$.
11. Define information gain; should we select features with high or low IG? (s.81, s.87) $IG=E(X)-E(X|Y)$, how much knowing the feature reduces uncertainty about the label; high IG = more entropy removed, so select high-IG features.
12. Compute the information gain of Humidity in the weather example. (s.82-86) $0.94-\frac{7}{14}0.592-\frac{7}{14}0.985=0.1515$.
13. Write the information gain of a term in text mining and explain the symbols. (s.88) See the formula and symbol table; it is the expected entropy reduction from partitioning the documents by presence/absence of the term.

Common confusions

• Slide idf vs Intro idf: this chunk writes $idf(w)=\log (1+N/df(w))$ (s.67), whereas 01 Intro s.26 writes $\log (N/df)$. Use the form given in the lecture you are asked about; with the “1+” version idf is never exactly 0.
• Presence vs frequency vs thresholded frequency: presence is 0/1, frequency is the raw count, thresholded frequency caps counts at 2 (“2 or more”) (s.66).
• High vs low entropy: high entropy = mixed classes / unpredictable / more information; low entropy = pure / predictable / less information (s.78-79). Do not confuse “high entropy” with “good split”: a good feature has high information gain, which produces low-entropy children.
• Entropy vs information gain: entropy is a property of one set; IG is the drop in entropy after splitting by a feature (parent entropy minus weighted child entropies).
• Hypernym vs hyponym; holonym vs meronym: hypernym = up to the superordinate (breakfast -> meal), hyponym = down to subtypes (meal -> lunch); meronym = has-part/has-member direction (table -> leg), holonym = part-of/member-of direction (course -> meal) (s.62).
• BoW storage vs BoW meaning: CSR only changes how the matrix is stored (pairs instead of zeros), not its content (s.70).
• Feature selection vs feature transformation: selection keeps a subset of the original words (s.73 strip diagram), transformation (PCA, topic models) creates new combined features (s.73).

Slide coverage

Slide	Content	Covered in
61	WordNet relations: breakfast / course browser entries	Detailed notes 1; Figures
62	Table of noun relations in WordNet	Detailed notes 1; Figures
63	Section title: Bag-of-Words: counting is everything	Detailed notes 2
64	Vector representation for documents (rows = docs, cols = features)	Detailed notes 2
65	BoW steps + cat/dog example (corpus, vocab, countVec)	Detailed notes 2; Figures (inconsistency flagged)
66	Document features: presence, frequency, thresholded	Detailed notes 2
67	tf-idf definition and idf formula	Detailed notes 3; Formulas
68	tf-idf intuitive logic	Detailed notes 3
69	Multiword features and association measure	Detailed notes 4; Formulas
70	Compressed Sparse Row representation	Detailed notes 5; Figures
71	BoW pros and cons	Detailed notes 6
72	Section title: Dictionary Reduction	Detailed notes 7
73	Dictionary reduction techniques, curse of dimensionality figure	Detailed notes 7; Figures
74	Feature selection by attribute ranking (flow diagram)	Detailed notes 7; Figures
75	Section title: Feature Selection based on Information Gain	Detailed notes 7b
76	Weather example table (features vs label)	Detailed notes 7b; Figures
77	Impurity of splits (20P/20N, three features)	Detailed notes 7b
78	Entropy definition and example	Detailed notes 7b; Formulas
79	Properties of entropy (max / min)	Detailed notes 7b; Formulas
80	Binary entropy and curve, [9+,5-] = 0.94	Detailed notes 7b; Figures
81	Entropy, conditional entropy, mutual information (IG)	Detailed notes 7b; Formulas
82	IG example: overall entropy 0.94	Detailed notes 7b; Figures
83	IG example: Humidity split formula	Detailed notes 7b; Figures
84	IG example: H_L for Humidity = Normal	Detailed notes 7b; Figures
85	IG example: H_L = 0.592, H_R = 0.985	Detailed notes 7b; Figures
86	IG example: IG(Humidity) = 0.1515	Detailed notes 7b; Formulas
87	Exercise: IG for outlook, temperature, windy; high or low IG?	Detailed notes 7b; Formulas (added context)
88	Information gain of a term in text mining	Detailed notes 7b; Formulas
89	Section title: Document-term Matrix & Cosine Similarity	Detailed notes 8
90	Troll 2 example sentences (video frame)	Detailed notes 8; Figures

Previous: 02 Pre-processing - Chunk 02 (s.31-60) | Next: 02 Pre-processing - Chunk 04 (s.91-120) | Up: 02 Pre-processing - Index