Text Preprocessing
Goal: clean and normalize text for downstream processing (01 Intro s.13).
- Simple: remove HTML tags/punctuation, lowercase, Concept - Tokenization.
- Advanced: Concept - Stopword Removal, Concept - Stemming / Concept - Lemmatization, Concept - POS Tagging, entity recognition, word sense disambiguation, synonym/pronoun normalization (s.13-14).
- Example (s.13): “Hello, I’m LOVING this NEW Python library! It’s amazing!!!” ->
hello love new python libari amaz. - Tools: RapidMiner Text Processing Extension (Tokenize -> Filter Stopwords -> Stem (Porter)) (s.20); sklearn
CountVectorizer, NLTKPorterStemmer(s.21).
Related
Concept - Text Mining Process | Concept - Bag of Words
Appears in
02 Pre-processing (s.1-30)
- Goal: turn text into numerical features (columns of a spreadsheet) usable by data-mining procedures; word occurrence is easy, grammatical function is hard (s.12). Motivated with a feature-extractor + Concept - Linear Model email example (Concept - Feature Extraction, s.4-10).
- Steps covered: collect documents (Concept - Corpus, s.13-14), Concept - Text Normalization (s.16), Concept - Tokenization (s.18-21), Concept - Stopword Removal (s.24-25), Concept - Case Folding (s.29), Concept - Stemming / Concept - Lemmatization (s.30). Background: Concept - Zipf’s Law, Concept - Tokens and Types, Concept - Word Sense Relations.
- Source: 02 Pre-processing - Chunk 01 (s.1-30)