Tokenization
Breaking text into single words or N-grams (01 Intro s.14). First step of Concept - Text Preprocessing (s.12). Example (s.13): “Hello, I’m LOVING…” -> hello i m loving this new python library it s amazing (note: apostrophes split “I’m” into i, m). RapidMiner operator: Tokenize (s.20).
Related
Concept - Stopword Removal | Concept - Stemming
Appears in
02 Pre-processing (s.17-21)
- Breaks the stream of characters into words/tokens; trivial for humans, hard for programs because some characters are delimiters only sometimes (s.18).
- White space (space, tab, newline) is always a delimiter and never a token; ( ) <> ! ? ” are always delimiters and may also be tokens (s.18).
- . , : - ’ depend on context: numbers (100,000; 333-1221), abbreviations (Dr.), within tokens (isn’t, D’angelo), possessive (Tess’); customize the tokenizer for the text (e.g. part: 123-4567); language-dependent (s.19).
- Open issues (s.20): Finland’s capital (Finland / Finlands / Finland’s?), What’re/I’m/isn’t (what are / I am / is not?), Hewlett-Packard, state-of-the-art, San Francisco (one token or two?). NLTK tokenizers (Treebank, WordPunct, Punkt, Whitespace, pattern) split “Finland’s” differently.
- Chinese/Japanese have no spaces between words; Japanese mixes Katakana, Hiragana, Kanji, Romaji (s.21).
- See Concept - Text Normalization, Concept - Tokens and Types. Source: 02 Pre-processing - Chunk 01 (s.1-30)