Concept - TF-IDF

TF-IDF

Term frequency-inverse document frequency: evaluates how important a word is to a corpus of documents (01 Intro s.26).

$w_{ij}=t{f}_{ij}\times id{f}_i,id{f}_i=\log \frac{N}{d{f}_i}$

$N$ = total docs in corpus; $d{f}_i$ = number of docs in which $t_i$ appears. More weight to rare words, less weight to common words (domain-specific stopwords). The ${\log }_2(x)$ curve (s.26) shows growth is dampened.

Added context

A term in all documents gets $idf=\log 1=0$.

Related

Concept - Bag of Words | Concept - Stopword Removal

Appears in

• 01 Intro - Chunk 01 (s.1-30)

In Naive Bayes (03 Text Classification, s.146-147)

Text features after preprocessing: word counts, frequencies / TF-IDF values, or binary indicators (s.146). Concept - Multinomial Naive Bayes accepts counts or TF-IDF values as input (s.147). See 03 Text Classification - Chunk 05 (s.121-150).

Worked example with cosine similarity (s.38-39) (01 Intro - Chunk 02 (s.31-48))

For d1 = “Saturn is the gas planet with rings.” (7 words) the TF-IDF vector starts $(\frac{1}{7}\log \frac{3}{2},\frac{1}{7}\log \frac{3}{3},\frac{1}{7}\log \frac{3}{3},\dots )$: is/the occur in all 3 docs so weight 0; rarer terms (with, rings: 0.07; Jupiter, largest in d2: 0.08) get the highest weights. Combined with Concept - Cosine Similarity it is a commonly used combination for text clustering (Concept - Document Clustering): cos(d1,d2)=0.13, cos(d1,d3)=0.05, cos(d2,d3)=0.00.

From 02 Pre-processing s.67-68

$\text{tf-idf}(w)=tf(w)\cdot idf(w)$ with $idf(w)=\log \left(1+\frac{N}{df(w)}\right)$ (note the “1+” here vs $\log \frac{N}{df}$ in 01 Intro). tf = word count scaled by an importance factor; idf looks at $df(w)$ (documents containing $w$) and reverses the scaling; $N$ = number of documents (s.67). Intuition: importance grows with times a word appears, shrinks with number of documents containing it; “the”, “I”, “on”, “document” get a scale near zero (s.68). See 02 Pre-processing - Chunk 03 (s.61-90).