Concept - Latent Semantic Analysis (LSA)

Definition

LSA (also called LSI in the slides, “reduced LSI”) represents documents in a reduced, dense topic space obtained by applying Concept - Singular Value Decomposition (SVD) to the document-term matrix and keeping only the most important dimensions (Concept - Truncated SVD). It tries to capture semantic information (documents talking about the same topic) and does not require documents to use the same words (s.190).

Intuition

Each latent “topic” is a linear combination of words (e.g. Topic 1 = -0.44 car - 0.70 gas - 0.26 oil - 0.47 road - 0.13 truck) (s.183). Dropping minor topics removes noise and lets different words share a dimension, so documents about the same thing become similar even without word overlap (s.186, s.191).

Recipe (s.192)

1. Document-term matrix $C$. 2. SVD $C=U\Sigma V^T$. 3. Corpus in reduced space: $U_d{\Sigma }_d$. 4. New doc as term vector $\mathbf{q}$. 5. Map: $\mathbf{q}{V}_d$ ($V_d=V[:,0:d]$, $d$ < rank, s.189).

Example (s.181-189)

6 docs x 5 terms (car, gas, oil, road, truck). With all 5 topics cosine(doc4, doc5) $\approx 0$; with 2 topics it is 0.75.

Why use it (s.190-191)

Addresses Concept - Synonymy and Semantic Relatedness; Eckart-Young theorem: SVD picks the least costly mapping.

Related

Concept - Singular Value Decomposition (SVD) | Concept - Truncated SVD | Concept - Synonymy and Semantic Relatedness | Concept - Cosine Similarity | Concept - Bag of Words | Concept - Other Document Representation Approaches

Source: 02 Pre-processing - Chunk 07 (s.181-193)

From chunk 06 (s.176-177)

LSA is the lecture’s chosen way to find the linear projection matrix ($m\times d$) that maps the $n\times m$ term-document matrix to an $n\times d$ document-”topic” matrix; the projection matrix is found by matrix decomposition (“Latent Semantic Analysis: SVD”, s.177). Alternative: Concept - Non-negative Matrix Factorization (NMF). See Concept - Dimensionality Reduction.

Source: 02 Pre-processing - Chunk 06 (s.151-180)