Text Normalization
Every NLP task, including text mining, needs text normalization (02 Pre-processing s.16):
- Segmenting/tokenizing words in running text (Concept - Tokenization)
- Normalizing word formats: convert to standard or common forms (Concept - Case Folding, Concept - Stemming, Concept - Lemmatization)
- Segmenting sentences in running text
Why (s.27): in IR, indexed text and query terms must have the same form (match U.S.A and USA). Define equivalence classes of terms. Alternative: query expansion (enter window -> search window, windows; enter windows -> search Windows, windows, window), which is potentially more powerful but less efficient.
Effect (s.28): reduces the number of distinct types and increases the frequency of some types (types, typed -> type); whether it is needed is application-dependent; can matter for frequency-based classifiers.
Related
Concept - Text Preprocessing | Concept - Tokens and Types