Zipf’s Law
Word frequencies in texts follow a power-law distribution (02 Pre-processing s.22): a small number of very frequent words and a big number of low-frequency words. Also called Zipf’s Law. Figure (s.22): steeply falling curve, x = Word Freq., y = Word Probs.; green head = few frequent words, long yellow tail = many rare words.
Motivates Concept - Stopword Removal (frequent function words) and why Concept - Stemming helps make statistics reliable (rare types get merged, s.30).
Related
Concept - Tokens and Types | Concept - Word Sense Relations