Concept - Pruning

Definition: pruning of document vectors specifies if and how too frequent or too infrequent words are ignored, to reduce the number of features (s.31, 01 Intro - Chunk 02 (s.31-48)).

Options (s.31)

• Percentual: ignore words appearing in less / more than a given percentage of all documents.
• Absolute: ignore words appearing in less / more than a given number of documents.

Tools (s.31)

• RapidMiner Process Documents from Files: prune method = absolute, prune below absolute = 30, prune above absolute = 250.
• Python: TfidfVectorizer(min_df=0.1, max_df=0.3) (percentual); TfidfVectorizer(min_df=5, max_df=20) (absolute).

Intuition

Very rare words (few documents) are unreliable features and blow up dimensionality; very frequent words behave like domain-specific stopwords and carry little discriminative information.

Related

Concept - Feature Selection, Concept - Stopword Removal, Concept - TF-IDF, Concept - Bag of Words