Concept - Text Mining Challenges

Why is text mining difficult? (s.46)

1. Many ways to represent similar concepts (space ship, flying saucer, UFO).
2. “Countless” combinations of subtle, abstract relationships among concepts (drugs and diseases).
3. High dimensionality (tens of hundreds of thousands of features).
4. Data variation: ImageNet exists for images, but no such huge labelled text data.
5. Ambiguity of language: word level (bank), sentence level (“I heard his cell phone in my office”).

Summary (s.47): the main challenge is preprocessing and vectorization (to apply known data mining algorithms); many alternative techniques, so experiment; focus shifted from bag-of-words to embeddings; OK-ish results are easily achieved.

Source: 01 Intro - Chunk 02 (s.31-48)

Related

Concept - Text Mining, Concept - Bag of Words, Concept - Word Embeddings, Concept - Feature Selection