Stopword Removal
- Stopwords: very frequent English words likely useless for text mining, e.g. the, of, and, to, an, is, that (01 Intro s.15). Typically ~400-500 words; a domain-specific list may be added.
- Why remove: (1) reduce data set size - stopwords are 20-30% of total word count; (2) improve effectiveness - they may confuse the mining algorithm (s.15).
- s.16: long example list (a, about, above … also, for, its, made, one, said, the, was, were…).
- Tools: RapidMiner Filter Stopwords (English) (s.20); sklearn
stop_words='english'(s.21). - Domain-specific stopwords also get low weight via Concept - TF-IDF (s.26).
Related
Concept - Text Preprocessing | Concept - Tokenization | Concept - Stemming
Appears in
02 Pre-processing (s.24-25)
- Stop words are words that, from a non-linguistic view, do not carry information; mainly a functional role; removed to help methods perform better (s.24).
- Natural-language dependent: English A, ABOUT, ABOVE, ACROSS, AFTER, FROM, AGAIN…; Chinese 的, 一, 不, 在, 有 (s.24).
- Example (s.25): in “Survey of Information Retrieval - guide to IR, with an emphasis on web-based projects” the stop words are of, to, with, an, on.
- Related: Concept - Zipf’s Law. Source: 02 Pre-processing - Chunk 01 (s.1-30)