Tokens and Types
- = number of tokens (running words)
- = vocabulary = set of types (distinct words); = size of the vocabulary (02 Pre-processing s.23)
| Corpus | Tokens = N | Types = |V| |
|---|---|---|
| Switchboard phone conversations | 2.4 million | 20 thousand |
| Shakespeare | 884,000 | 31 thousand |
| Google N-grams | 1 trillion | 13 million |
Normalization (Concept - Stemming, Concept - Lemmatization) reduces the number of types and raises the frequency of the remaining ones (s.28, s.30).
Related
Concept - Zipf’s Law | Concept - Corpus | Concept - Text Normalization