01 · Intro to Text & Web Mining

What → process → preprocessing → features → selection → clustering → classification → challenges

(s.N) = slide number.

Text mining: extraction of implicit, previously unknown, potentially useful information from large amounts of textual resources (s.4)

Search vs discovery (s.10) IR = search, goal-oriented (text) Text mining = discovery, opportunistic (text) Data mining = discovery on structured data

Applications (s.5-9) news classification + clustering, spam filtering, sentiment analysis, search auto-completion / query topics, information extraction

Preprocessing: clean + normalize text. Simple: remove HTML/punctuation, lowercase, tokenize. Advanced: stopwords, stemming/lemmatization, POS, entities (s.13-14)

Stemming: User, users, used, using → use; cuts term vector by 40-50% (s.17). Rules: drop -s, -es, -ing, -ed; ies → y (s.18)

Stemming vs lemmatization (s.19) Stem: rule-based, fast, may be non-word (amazing → amaz; Porter) Lemma: dictionary + POS, slower, valid word (amazing → amaze; WordNet)

~90% of the world's data is unstructured (web pages, emails, papers); only 10% structured (s.3)

Information extraction subtasks: NER + disambiguation, relationship extraction, fact extraction (s.9)

Process (s.11): preprocessing → feature generation → feature selection → data mining / pattern discovery → interpretation / evaluation; dotted feedback arrows back to the start (iterative)

↪ stage 1 → 03 · Preprocessing

Main challenge (s.47): preprocessing and vectorization so that well-known data mining algorithms apply. Experiment; focus moved from bag of words to embeddings

Example (s.13): 'Hello, I'm LOVING this NEW Python library!' → hello love new python libari amaz. POS tagging: John (noun) gave (verb) the (det) ball (noun) (s.14)

↪ then → 04 · Feature generation

Bag of words: document = vector of word counts, order ignored. Pros: simple, interpretable, fast. Cons: loses order, high dimensionality, sparse (s.23-24)

↪ followed by → 05 · Feature selection

4 vector types (s.25) binary term occurrence · term occurrence (bad for different lengths) · term frequency = count / words in doc · TF-IDF

TF-IDF (s.26) $w_{ij}=tf_{ij}\times idf_i$, $idf_i=\log\frac{N}{df_i}$ rare words up, common words down; term in all docs → idf = 0

↪ used in → 06 · Document clustering

Stopwords: ~400-500 words (the, of, and...); 20-30% of word count; removal shrinks data and helps algorithms (s.15)

Goal (s.34): given documents + similarity measure, clusters with high intra-, low inter-cluster similarity. Uses: news topics, email threads, document versions

Cosine (s.37) $\cos=\frac{d_1\bullet d_2}{|d_1||d_2|}$ for TF / TF-IDF vectors example $d_1,d_2$ → 0.315

Cosine + TF-IDF (s.38-39): common combination; is, the get idf = 0 → sims 0.13 / 0.05 / 0.00 (d2-d3 share only zero-weight terms)

Word embeddings: dense real vectors, 50-300 dims; related words close (dog, puppy); distributional similarity; Word2Vec skip-gram. Others: GloVe, fastText, BERT (s.28-29)

↪ used in → 06 · Document clustering

Sentiment analysis (s.42): supervised classification on pairs <text; sentiment>. Labels expensive; reviews' star rating = label (Amazon 142M, WebDataCommons 70M) (s.43)

Jaccard (s.35-36) $\frac{M_{11}}{M_{01}+M_{10}+M_{11}}$ for binary term occurrence; ignores 0-0 example 0.44 / 0.27 / 0.18

Embedding similarity (s.40): doc2vec or average word vectors, then cosine, word mover's distance or RNN/LSTM. Restaurant vs Eatery 0.723, vs Resource 0.146

One-hot vs distributed: one-hot = single 1 in V-dim vector, no similarity; distributed = dense, similarity meaningful (s.28)

Document classification (s.41): labeled training set → model class = f(features) → classify unseen docs. Methods: naive Bayes, SVM, RNN/LSTM, also KNN, random forest

Tricks (s.45): top-k words by correlation with label; sentiment lexicons (Bing Liu, AFINN)

Challenges (s.46): (1) many ways to express a concept (UFO, flying saucer); (2) subtle abstract relations; (3) high dimensionality; (4) little labelled text data vs ImageNet; (5) ambiguity (bank; 'heard his cell phone in my office')

Exam traps: short 'discuss / describe / explain' answers (see Past Exam note)

Term occurrence vs term frequency: count vs count / doc length (s.25)

Stem vs lemma: libari/amaz are not words; amaze is (s.13, 19)

Feature selection: not all features help; high-dimensional data hurts learners (s.30)

Pruning (s.31): ignore too frequent / too infrequent words. Percentual (share of docs) or absolute (number of docs). sklearn min_df/max_df: float = share, int = count

POS filter (s.32): adjectives (JJ) for sentiment; nouns for clustering. PENN tags (English), STTS (German); regex conditions

↪ helps → 07 · Classification & sentiment

Pruning ≠ stopword removal: fixed list vs document-frequency thresholds (s.31)

IR vs text mining: goal-oriented search vs opportunistic discovery (s.10)

Jaccard vs cosine: binary + ignores 0-0 vs weighted TF/TF-IDF (s.35, 37). Slide calls Jaccard 'dist' but it is a similarity

Clustering vs classification: similarity measure, no labels vs labeled training set (s.34, 41). Slide s.37 prints $\sqrt6=2.245$ (typo; 2.449)

Preprocessing for sentiment (s.44): keep !, ?!, smileys, markup, CAPS; replace with words (smiley → great, COOL → cool cool)