Lemmatization
Linguistic approach to reduce words to root form: uses a dictionary & POS tagging; slower but more accurate; produces valid dictionary words. Tool: NLTK WordNet Lemmatizer. Example: amazing -> amaze (01 Intro s.19).
| Concept - Stemming | Lemmatization | |
|---|---|---|
| Approach | rules, strip suffixes | dictionary + POS |
| Speed | fast | slower |
| Output | may be non-words (amaz) | valid words (amaze) |
Related
Concept - POS Tagging | Concept - Text Preprocessing
Appears in
02 Pre-processing (s.28, s.30)
- The lecture titles s.30 “Lemmatization - Stemming to a Root”: converts to a root form without inflectional or derivational prefixes/suffixes; words with the same core meaning are coalesced (e.g. “reapplied”, “applications” -> “apply”); fewer types, more reliable distributional statistics.
- Inflectional suffixes keep the word’s meaning (different grammatical forms); derivational suffixes change the meaning (s.30).
- See Concept - Stemming, Concept - Text Normalization. Source: 02 Pre-processing - Chunk 01 (s.1-30)
From 02 Pre-processing, s.31
- Reduces variant forms to the base form: am/are/is -> be; car, cars, car’s, cars’ -> car; “the boy’s cars are different colors” -> “the boy car be different color”.
- Must find the correct dictionary headword (root / lemma). Stanford CoreNLP supports it: denormalization -> denormalization, reapplications -> reapplication, reapplied -> reapply.
- See Concept - Morphology. Source: 02 Pre-processing - Chunk 02 (s.31-60)