Concept - Out of Vocabulary (OOV)

Definition

The Out of Vocabulary (OOV) problem arises when a test word is not present in the training dataset at all (s.31). It differs from the case where a word appears in some classes but not all, which is solved with Concept - Additive Smoothing (s.31).

Two ways to handle it (s.31)

Option	Details
Ignore OOV words	Calculations can still be made without them. If all words are OOV, Naive Bayes reverts to the class priors (Concept - Prior Probability).
<UNK> token	Introduce a new token <UNK> for unknown words and replace all OOVs with it. The vocabulary then has $\Vert V\Vert +1$ tokens, and smoothing must be applied to <UNK> as well.

flowchart TD
    A[Test word not in training vocabulary] --> B{Handle}
    B --> C[Ignore the word]
    B --> D[Replace by UNK token]
    C --> E[All words OOV -> only priors remain]
    D --> F[Vocabulary V+1, smooth UNK]

Related

Concept - Zero Probability Problem, Concept - Additive Smoothing, Concept - Naive Bayes, Concept - Multinomial Naive Bayes

Source chunk: 03 Text Classification - Chunk 02 (s.31-60)