Concept - Word Embeddings

Word Embeddings

• Represent each word as a dense vector of real numbers (distributed representation, e.g. 50-300 numbers) instead of one-hot (01 Intro s.28).
• Semantically related words (dog, puppy) are close in vector space -> handle synonyms/related terms better than Concept - Bag of Words.
• Based on distributional similarity: similar words appear in similar contexts. Skip-gram (Word2Vec): predict context words of each word with a neural net; the hidden layer $\mathbf{h}\in {\mathbb{R}}^N$ is the embedding.
• Methods: Word2Vec (Google), GloVe (Stanford), fastText (Facebook AI Research; 294 languages), BERT (Google). GloVe pretrained on Common Crawl, Wikipedia, Tweets (s.29). Tools: Python Gensim, RapidMiner Word2Vec extension.

Related

Concept - Text Mining Process

Appears in

• 01 Intro - Chunk 01 (s.1-30)

Embedding-based document similarity (s.40) (01 Intro - Chunk 02 (s.31-48))

Translate documents into vectors (doc2vec, or average of word embeddings), then compare with cosine similarity, word mover’s distance or neural nets (RNNs/LSTMs). Demo: Restaurant-Eatery 0.723 vs Restaurant-Resource 0.146. Libraries: gensim (doc2vec, word2vec), RapidMiner word2vec extension. Summary s.47: the field’s focus has shifted from bag-of-words to embeddings. See Concept - Document Clustering.