Stemming

Finding the stem of a word: User, users, used, using -> use; Engineering, engineered -> engineer (01 Intro s.17).

Benefits: better effectiveness, matching similar words, smaller term vector (combining same-stem words reduces it by 40-50%) (s.17).
Rules (s.18): ends with consonant other than s + s -> delete s; ends in es -> drop s; ends in ing -> delete ing (unless remainder is a single letter); ends in ed preceded by consonant -> delete ed (unless single letter remains); “ies” (not “eies”/“aies”) -> “y”.
Rule-based, fast, may produce non-words (amazing -> amaz, library -> libari). Tool: Porter Stemmer (NLTK) (s.19, s.21).
Contrast with Concept - Lemmatization.

Appears in

Normalization converts tokens to a standard form via stemming or lemmatization; application-dependent; reduces the number of types and increases the frequency of some types (types, typed -> type) (s.28).
s.30 (“Lemmatization - Stemming to a Root”): reduce to a root with no inflectional (-ed, -ing, -s; different grammatical forms) or derivational (-ism, -ful, -fy; change meaning) affixes. E.g. “denormalization” -> “norm”; “reapplied”, “applications” -> “apply”. Aggressive stemming drastically reduces types, making distributional statistics more reliable.
See Concept - Text Normalization, Concept - Tokens and Types. Source: 02 Pre-processing - Chunk 01 (s.1-30)

Inflectional stemming = regularising grammatical variants (singular/plural, present/past); called morphological analysis (s.33). English needs part rules, part dictionary (sought -> seek); token-only stemmers err by ambiguity (bored: adjective vs past of bore) (s.33).
Stemming = crude chopping of affixes, language dependent; automates/automatic/automation -> automat (s.34). Example: “for example compressed and compression are both accepted as equivalent to compress” -> “for exampl compress and compress ar both accept as equival to compress” (s.34).
NLTK demo (s.35): Porter -> “Stem is funnier than a bummer say the sushi love comput scientist”; Lancaster (more aggressive) -> “stem is funny than a bum say the sush lov comput sci”.
See Concept - Morphology. Source: 02 Pre-processing - Chunk 02 (s.31-60)