Concept - Bag of Words

Bag of Words (BoW)

Each document is a bag of words/terms: each term is a feature, order is ignored, each document is a vector (01 Intro s.23, s.25). Collected in a term-document matrix (s.23-24).

Vector creation techniques (s.25):

1. Binary term occurrence - appears or not.
2. Term occurrence - raw count (problematic for different-length docs).
3. Term frequency - count / number of words in document.
4. Concept - TF-IDF.

• Pros: simple, interpretable, fast. Cons: loses word order, high dimensionality (s.23).
• sklearn: CountVectorizer(binary=True/False), TfidfVectorizer(use_idf=False/True); RapidMiner “vector creation” parameter (s.27).
• Alternative: Concept - Word Embeddings.

Related

Concept - Feature Selection | Concept - Text Preprocessing

Appears in

• 01 Intro - Chunk 01 (s.1-30)

From 02 Pre-processing (s.99-105)

Worked mini-example: phrases “Hello, World!” and “Hello!” become count rows over the vocabulary {Hello, World}: (1,1) and (1,0) (s.99-105); such count vectors can be plotted and compared with Concept - Cosine Similarity (s.106-111). See 02 Pre-processing - Chunk 04 (s.91-120).

Used for similarity (02 Pre-processing)

BoW count vectors can be compared with Concept - Cosine Similarity: the angle between the count vectors of two phrases (identical = $0^{\circ }$ -> 1; no shared words = $90^{\circ }$ -> 0) (02 Pre-processing s.121-150). See 02 Pre-processing - Chunk 05 (s.121-150).

From 03 Text Classification chunk 13 (s.367, 372)

Bag-of-words is the typical sentiment classifier input (s.366) but misses negation and contrast (s.367); in WSD the context words serve as bag-of-words features paired with the sense label (s.372). See 03 Text Classification - Chunk 13 (s.361-381).

In Naive Bayes (03 Text Classification, s.137-150)

Naive Bayes “treats language like a bag full of words and each message is a random handful of them” (s.142), so “Dear Friend” and “Friend Dear” both score 0.08 (s.138-140). Bag-of-Words vectors feed Concept - Multinomial Naive Bayes (counts, s.147) while binary versions feed Concept - Bernoulli Naive Bayes (s.148, s.150). See 03 Text Classification - Chunk 05 (s.121-150).

From 02 Pre-processing chunk 06 (s.170-172)

Bag-of-words (TF-IDF) yields a Concept - Document-Term Matrix of shape $n\times m$ ($n$ docs, $m$ vocabulary), which is high-dimensional and sparse (s.171). It is “just counting, no semantic”: related terms such as gas/oil in different documents have cosine similarity 0 (s.172). Remedies: Concept - Feature Selection, Concept - Dimensionality Reduction (LSA/SVD).

Source: 02 Pre-processing - Chunk 06 (s.151-180)

From 02 Pre-processing s.63-71

• Steps: build vocabulary (all corpus words), count occurrences per document; rows = documents, columns = features (s.64-65). Example: “The cat and the dog play” / “The cat is on the mat” (s.65; second count row printed on the slide looks inconsistent).
• Entry values: presence (0/1), frequency, thresholded frequency (0, 1, 2+) (s.66), or Concept - TF-IDF (s.67).
• Extensions: Concept - Multiword Features (s.69); storage as Concept - Sparse Matrix (CSR) (s.70).
• Pros: simple, surprisingly effective, fast. Cons: order does not matter, no syntactic/semantic information, high dimensionality (s.71) -> Concept - Dictionary Reduction, Concept - Curse of Dimensionality.
• See 02 Pre-processing - Chunk 03 (s.61-90).