Concept - Multiword Features

Multiword Features

Features that are phrases rather than single words, e.g. frequent n-grams such as “text mining”, “hip hop” (02 Pre-processing s.69).

Association Measure for a multiword $T$ ($size(T)$ = number of words in $T$; $freq(T)$ = times $T$ occurs in the collection):

$AM(T)=\frac{size(T){\log }_{10}(freq(T))freq(T)}{{\sum }_{wor{d}_i\in T}freq(wor{d}_i)}$

Multiword features are generally not frequent in a collection, but when they occur they are often highly predictive (s.69).

Added context

Intuition for the formula: the numerator rewards long, frequent phrases; the denominator (sum of the individual word frequencies) penalizes phrases whose words are common on their own. The slide does not explain this.

Related

Concept - Bag of Words | Concept - Feature Selection | Concept - Tokenization

Appears in

• 02 Pre-processing - Chunk 03 (s.61-90)