01 Intro - Chunk 01 (s.1-30)

Overview

This part of the lecture (DAT402 Text and Web Mining, slides adapted from Christian Bizer, Univ. of Mannheim, and Zhao Rui, NTU) motivates Concept - Text Mining: about 90% of the world’s data is unstructured (s.3). It defines text mining, lists typical applications (classification, spam filtering, sentiment analysis, clustering, search-log mining, information extraction), and contrasts it with data mining and information retrieval (s.4-10). It then walks through the five-stage Concept - Text Mining Process: Concept - Text Preprocessing (tokenization, stopwords, stemming, POS tagging), feature generation (Concept - Bag of Words, Concept - TF-IDF, Concept - Word Embeddings) and the start of Concept - Feature Selection (s.11-30), with RapidMiner and Python (sklearn / NLTK) examples.

Detailed notes

1. Motivation and definition (s.1-4)

Course: DAT402 Text and Web Mining – Introduction; slides taken from Christian Bizer (Mannheim) and Zhao Rui (NTU) (s.1).
Outline: 1. What is Text Mining? 2. Text Preprocessing 3. Feature Creation 4. Feature Selection 5. Pattern Discovery (s.2).
Motivation: approximately 90% of the world’s data is held in unstructured formats (source: Oracle Corporation); only 10% is structured (pie chart) (s.3).
- Examples of unstructured / semi-structured data: web pages, emails, customer complaint letters, corporate documents, scientific papers, books in digital libraries (s.3).
Definition – Text Mining: “The extraction of implicit, previously unknown and potentially useful information from large amounts of textual resources.” (s.4)
Text mining sits at the intersection of: Data Mining, Information Retrieval, Computational Linguistics & NLP, Statistics, Machine Learning (s.4).

2. Text mining applications (s.5-10)

Slide 5 lists: (1) classification of news stories, (2) email and news filtering / SPAM detection, (3) sentiment analysis, (4) clustering of documents or web pages, (5) search term auto-completion, (6) information extraction.

Application	What it does	Technique / example	Slide
News aggregation (Google News)	Groups related stories and assigns them to categories	Mixture of document clustering (stories grouped) and classification (sections like World, U.S., Business, Technology, Sports…)	s.6
Concept - Sentiment Analysis	Determine polarity of a text at document, sentence or feature/aspect level	Polarity values: positive / neutral / negative, or Likert scale (1 to 10). Document level: tweets about politicians. Aspect level: product reviews	s.7
Search log mining	Analysis of search queries issued by large user communities	(1) Search-term auto-completion using association analysis; (2) query topic detection using classification	s.8
Concept - Information Extraction	Automatically extract structured information from unstructured / semi-structured documents	Subtasks: Named Entity Recognition and Disambiguation; Relationship Extraction; Fact Extraction	s.9

Information extraction subtasks with the slide’s examples (s.9):
- Named Entity Recognition and Disambiguation: “The parliament in Berlin has decided …” -> Which parliament? Which Berlin?
- Relationship Extraction: PERSON works for ORGANIZATION; PERSON located in LOCATION.
- Fact Extraction: CITY has population NUMBER; COMPANY has turnover NUMBER [Unit].

3. Search versus Discovery (s.10)

2x2 grid: rows = data type (structured data / text); columns = goal (Search/Query = goal-oriented vs Discovery = opportunistic).

	Search / Query (goal-oriented)	Discovery (opportunistic)
Structured data	Query Processing	Data Mining
Text	Information Retrieval	Text Mining (highlighted in red)

4. The Text Mining Process (s.11, s.30)

Four numbered steps on the slide (s.11):

Text Preprocessing - syntactic and/or semantic analysis
Feature Generation - bag of words, word embeddings
Feature Selection - reduce large number (of features)
Data Mining - clustering, classification, association analysis Then Interpretation / Evaluation. Dotted feedback arrows lead from every stage back to the start (iterative process). See the diagram below.

5. Text Preprocessing (s.12-21)

Stage list (s.12): 1. Tokenization 2. Stopword Removal 3. Stemming 4. POS Tagging.

Goal: clean and normalize text for downstream processing (s.13).
Simple preprocessing (s.13): remove HTML tags and punctuation; normalize case (lowercase); tokenization (split into words).
Advanced preprocessing (s.13): stop word removal; stemming / lemmatization; POS tagging, entity recognition.
Concrete example (s.13):
- Raw: “Hello, I’m LOVING this NEW Python library! It’s amazing!!!”
- After tokenize & lowercase: hello i m loving this new python library it s amazing
- After remove stopwords & stem: hello love new python libari amaz -> “Ready for feature generation!”
- (Note the stemmer output: library -> libari, amazing -> amaz; stems need not be real words.)
Simple syntactic processing (s.14): text cleanup (remove punctuation and HTML tags); Concept - Tokenization (break text into single words or N-grams).
Advanced linguistic processing (s.14):
- Word Sense Disambiguation - determine which sense a word has; normalize synonyms (United States, USA, US); normalize pronouns (he, she, it).
- Concept - POS Tagging - parse sentences according to grammar, determine function of each term, e.g. John (noun) gave (verb) the (det) ball (noun).

Stopword removal (s.15-16)

Many of the most frequently used English words are likely useless for text mining -> stopwords; examples: the, of, and, to, an, is, that.
Typically ~400 to 500 such words; a domain-specific stopword list may be added for an application (s.15).
Why remove? (1) Reduce data set size - stopwords account for 20-30% of total word count. (2) Improve effectiveness of mining methods - stopwords may confuse the algorithm (s.15).
s.16 shows a long alphabetical list of English stopwords (a, about, above, across, after … you’ll); a few are highlighted in red larger font: also, for, its, made, one, said, the, was, were. (Takeaway: stopwords include articles, prepositions, pronouns, auxiliary verbs, plus fillers like “also”, “said”).

Stemming (s.17-19)

Technique to find the stem of a word: User, users, used, using -> use; Engineering, engineered -> engineer (s.17).
Usefulness: improves effectiveness of mining methods; matches similar words; reduces term-vector size - combining words with the same stem may reduce the term vector by 40-50% (s.17).
Basic stemming rules (s.18) - “remove endings”:
- if a word ends with a consonant other than s, followed by an s, delete the s;
- if a word ends in es, drop the s;
- if a word ends in ing, delete the ing unless the remaining word consists only of one letter (or of “th” - text cut off on slide);
- if a word ends with ed, preceded by a consonant, delete the ed unless this leaves only a single letter;
- and more (”…”).
- “transform words”: if a word ends with “ies” but not “eies” or “aies” then “ies -> y”.
Stemming vs. Lemmatization (s.19): both reduce words to root form but differ in approach and accuracy.

	Stemming	Lemmatization
Approach	Rule-based: removes suffixes (-ing, -ed, -s)	Linguistic: uses dictionary & POS tagging
Speed	Fast, simple rules	Slower
Accuracy / output	May produce non-words	More accurate, produces valid dictionary words
Tool on slide	Porter Stemmer (Python NLTK)	NLTK WordNet Lemmatizer
Example	amazing -> amaz	amazing -> amaze

See Concept - Stemming and Concept - Lemmatization.

Tools (s.20-21)

RapidMiner (s.20): requires installing the Text Processing Extension. Operators Process Documents from Files and Process Documents from Data; inside, a nested subprocess chains Tokenize -> Filter Stopwords (English) -> Stem (Porter).
Python (s.21):
- sklearn: CountVectorizer(analyzer='word', stop_words='english'), load_files('directory_of_files', encoding='utf-8'), matrix = vectorizer.fit_transform(docs).
- NLTK: PorterStemmer().stem(item) applied in a loop to tokens ['Jupiter','is','the','largest','gas','planet'].
- Referenced docs: scikit-learn “working with text data” tutorial; NLTK book ch.3.

6. Feature Generation (s.22-29)

Section heading (s.22): 3. Feature Generation - 1. Bag-of-Words, 2. Word Embeddings.

Bag-of-Words (s.23-27)

Documents are represented as vectors counting word occurrences, ignoring order (s.23).
Example corpus (s.23): Doc 1 “Saturn is a gas planet”; Doc 2 “Jupiter is a gas planet”; Doc 3 “Saturn is the Roman god of sowing”.
Term-Document matrix on the slide (terms: Saturn, is, a, gas, planet, Jupiter, the, Roman, god, sowing):

	Saturn	is	a	gas	planet	Jupiter	the	Roman	god	sowing
Doc1	1	1	1	1	1	0	0	0	0	0
Doc2	0	1	1	1	1	1	0	0	0	0
Doc3	1	1	0	0	0	0	1	1	1	1

Pros: simple, interpretable, fast. Cons: loses word order, high dimensionality (s.23).
s.24 shows a real Term-Document Matrix: rows = stemmed terms (oil, price, opec, mln, market, barrel, bpd, dlrs, crude, saudi, kuwait, offici, meet, pct, product, accord, futur, minist, govern, month, report, sheikh, industri, produc, quota, reserv, world …), columns = documents A-T, with row sums (e.g. oil 85, price 63, opec 47) and column sums (total 2039). Column R is highlighted in blue (document R, total 44: oil 3, price 3, barrel 1, dlrs 5, crude 2, rest 0). Takeaway: the matrix is sparse (many zeros) and terms are stems (indicating prior stemming/stopword removal).
Feature generation via BoW (s.25): each word/term becomes a feature; order ignored; each document is a vector. Four vector-creation techniques:
1. Binary Term Occurrence - Boolean attribute: does the term appear or not.
2. Term Occurrence - number of occurrences of the term in the document (problematic if documents have different length).
3. Term Frequency - frequency = number of occurrences / number of words in document.
4. TF-IDF - next slide.
TF-IDF (s.26): evaluates how important a word is to a corpus of documents. $w_{ij} = t f_{ij} \times i d f_{i}$ and $i d f_{i} = lo g \frac{N}{d f _{i}}$ , with $N$ = total number of docs in corpus, $d f_{i}$ = number of docs in which term $t_{i}$ appears. Gives more weight to rare words and less weight to common words (domain-specific stopwords). A plot of $y = lo g_{2} (x)$ is shown (points at x=1,2,4,8 giving y=0,1,2,3): the log dampens growth, and $i df = 0$ when a term occurs in all docs.
In tools (s.27): RapidMiner Process Documents from Files operator: (1) specify files to process (text directories, file pattern), (2) select feature generation method via vector creation: TF-IDF, Term Frequency, Term Occurrences, Binary Term Occurrences. Python:
- CountVectorizer(binary=True) -> binary term occurrence
- CountVectorizer(binary=False) -> term occurrence
- TfidfVectorizer(use_idf=False) -> term frequency
- TfidfVectorizer(use_idf=True) -> TF-IDF

Word Embeddings (s.28-29)

Embeddings represent each word not as a single number in a word vector (one-hot representation) but as a vector of real numbers (distributed representation), e.g. 50 to 300 numbers (s.28).
Chosen so that semantically related words (dog, puppy) end up at similar locations in the vector space -> embeddings handle synonyms and related terms better than bag-of-words (s.28).
Calculated based on the assumption that similar words appear in similar contexts (distributional similarity). Skip-gram approach used by Word2Vec: predict context words for each word using a neural net (s.28).
Figures (s.28): top - country->capital relation lines (Spain-Madrid, Italy-Rome, Germany-Berlin, Turkey-Ankara, Russia-Moscow, Canada-Ottawa, Japan-Tokyo, Vietnam-Hanoi, China-Beijing) showing parallel offsets in vector space; bottom - skip-gram network: one-hot input $x \in R^{V}$ , weight matrix $W \in R^{V \times N}$ , hidden layer $h \in R^{N}$ (the word embedding, red arrow), then $W^{'} \in R^{N \times V}$ to $C$ softmax outputs $y_{1} .. y_{C} \in R^{V}$ (the context words).
Methods & pretrained models (s.29): Word2Vec (Google), GloVe (Stanford NLP Group), fastText (Facebook AI Research), BERT (Google). Pretrained embeddings can be downloaded: GloVe trained on Common Crawl, Wikipedia and Tweets; fastText embeddings for 294 languages. Usage: Python Gensim offers a Word2Vec implementation; RapidMiner has a Word2Vec extension on the marketplace. The slide shows the 50-dimensional GloVe embedding of the word “the” (pretrained on Common Crawl): a list of 50 real numbers (e.g. 0.418, 0.24968, -0.41242, …).

7. Feature Selection (s.30)

Start of section “4. Feature Selection”: Not all features help! Learners might have difficulty with high-dimensional data. The pipeline figure circles the Feature Selection stage (details continue in the next chunk).

Figures & diagrams

s.3 Pie chart: 10% structured vs ~90% unstructured / semi-structured data; takeaway: most data is text-like.
s.4 Venn-style petals: Text Mining overlaps Data Mining, Information Retrieval, ML, Statistics, Computational Linguistics & NLP.
s.6 Screenshot of Google News: left sidebar categories (World, U.S., Business, Technology, Entertainment, Sports, Science, Health, Spotlight) boxed in red = classification; clustered related story links (“Opinion”, “In Depth”) boxed in red = clustering.
s.7 Twitter sentiment gauges for five Irish politicians (e.g. Enda Kenny 51% of tweets, Negative; Gerry Adams 23% slightly negative; Eamon Gilmore 9% neutral; Micheal Martin 8%, John Gormley 6%) = document-level; “Apple iPod Touch 8gb - Top 5 Features” bars (picture 9.7, clarity 9.1, display 9.0, technology 9.0, navigation 8.7) = aspect-level.
s.8 Google auto-complete for “Samsung g” (samsung galaxy, galaxy s3, galaxy note, galaxy note 2) and Google Flu Trends world map (Australia: flu activity low) = query topic detection.
s.9 Screenshots of Google Squared (table of small dog breeds with weight, height, group) and DBpedia faceted browsing of skyscrapers (filters: type, floors, location, year) = structured data extracted from text.
s.10 2x2 grid Search vs Discovery (see above).
s.11 / s.30 Pipeline: Text -> Text Preprocessing -> Text Transformation (Feature Generation) -> Feature Selection -> Data Mining / Pattern Discovery -> Interpretation / Evaluation, with dotted feedback arrows back to the start; s.30 circles Feature Selection.
s.16 Wall of stopwords with highlighted words.
s.20 RapidMiner operator screenshots (Tokenize -> Filter Stopwords (English) -> Stem (Porter)).
s.21 Code listings for sklearn and NLTK.
s.24 Real term-document matrix (terms x documents A-T, with sums).
s.26 $lo g_{2} (x)$ curve and TF-IDF formulas.
s.27 RapidMiner parameter panel + Python code for vectorizer options.
s.28 Country-capital vector offsets and skip-gram network diagram.
s.29 GloVe-50 vector of “the”.

flowchart LR
    T[Text] --> P[1. Text Preprocessing<br/>tokenize, stopwords, stem, POS]
    P --> F[2. Feature Generation<br/>BoW, TF-IDF, embeddings]
    F --> S[3. Feature Selection<br/>reduce dimensionality]
    S --> D[4. Data Mining<br/>clustering, classification, association]
    D --> I[Interpretation / Evaluation]
    I -. feedback .-> T

flowchart TD
    A["Raw: Hello, I'm LOVING this NEW Python library! It's amazing!!!"] --> B["Tokenize + lowercase<br/>hello i m loving this new python library it s amazing"]
    B --> C["Remove stopwords + stem<br/>hello love new python libari amaz"]
    C --> D[Feature generation]

Key terms

Formulas

TF-IDF weight: $w_{ij} = t f_{ij} \times i d f_{i}$ (s.26)
Inverse document frequency: $i d f_{i} = lo g \frac{N}{d f _{i}}$ , $N$ = number of docs in corpus, $d f_{i}$ = number of docs containing $t_{i}$ (s.26)
Term frequency (s.25): $t f_{ij} = \frac{occurrences of term i in doc j}{number of words in doc j}$

Likely exam questions

Define text mining. (s.4) The extraction of implicit, previously unknown and potentially useful information from large amounts of textual resources.
Why is text mining important? (s.3) ~90% of the world’s data is unstructured/semi-structured (web pages, emails, documents, papers).
Discuss the difference between information retrieval and text mining. (s.10) IR = search/query, goal-oriented, on text; text mining = discovery, opportunistic, on text. Data mining is the discovery counterpart for structured data; query processing the search counterpart.
List the steps of the text mining process. (s.11) Text preprocessing, feature generation, feature selection, data mining (clustering/classification/association), then interpretation/evaluation.
Name text mining applications. (s.5) News classification, spam filtering, sentiment analysis, clustering, search auto-completion, information extraction.
What are the subtasks of information extraction? (s.9) Named entity recognition & disambiguation, relationship extraction, fact extraction.
Why remove stopwords? (s.15) Reduces data size (20-30% of word count) and improves effectiveness (they may confuse the mining algorithm).
Compare stemming and lemmatization. (s.19) See comparison table (rule-based/fast/may produce non-words vs dictionary+POS/slower/valid words).
Explain the bag-of-words model and its pros/cons. (s.23, s.25) Documents as vectors of term counts, order ignored; simple, interpretable, fast; loses word order, high dimensionality.
Explain TF-IDF and why it is used. (s.26) $t f \times lo g (N / df)$ ; upweights rare words, downweights common ones (domain-specific stopwords).
How do word embeddings differ from one-hot / BoW? (s.28) Dense real vectors of 50-300 dims, related words close together, based on distributional similarity; handle synonyms better.

Common confusions

Stem vs lemma: a stem may not be a word (library -> libari, amazing -> amaz); a lemma is a valid word (amazing -> amaze) (s.13, s.19).
Term occurrence vs term frequency: occurrence = raw count (bad for different-length documents); frequency = count / words in document (s.25).
Binary vs count vectors in sklearn: CountVectorizer(binary=True) = binary occurrence; TfidfVectorizer(use_idf=False) = term frequency (s.27).
Search vs discovery: IR is goal-oriented (you know what you search for); text mining is opportunistic (find unknown patterns) (s.10).
One-hot vs distributed representation: one-hot = single 1 in a V-dimensional vector (no similarity); distributed = dense real vector where similarity is meaningful (s.28).
IDF of common words: a term in every document has $i df = lo g (1) = 0$ and thus no weight (s.26, derived from formula).

Added context

The statement that $i df = 0$ for a term occurring in all documents follows directly from the formula on s.26; the slide only says common words get less weight.

Slide coverage

Slide	Content	Covered in
1	Title: DAT402 Text and Web Mining - Introduction (credits)	Detailed notes 1
2	Outline (5 topics)	Detailed notes 1
3	Motivation: 90% unstructured data	Detailed notes 1; Figures
4	Definition of text mining + related fields	Detailed notes 1; Figures
5	List of applications	Detailed notes 2
6	Google News: clustering + classification	Detailed notes 2; Figures
7	Sentiment analysis	Detailed notes 2; Figures
8	Search log mining	Detailed notes 2; Figures
9	Information extraction	Detailed notes 2; Figures
10	Search vs Discovery grid	Detailed notes 3
11	Text mining process	Detailed notes 4; Figures
12	Section: text preprocessing (4 steps)	Detailed notes 5
13	Preprocessing goal, simple/advanced, example	Detailed notes 5
14	Syntactic and linguistic preprocessing	Detailed notes 5
15	Stopword removal	Detailed notes 5
16	More stopword examples	Detailed notes 5; Figures
17	Stemming	Detailed notes 5
18	Basic stemming rules	Detailed notes 5
19	Stemming vs lemmatization	Detailed notes 5
20	Preprocessing in RapidMiner	Detailed notes 5
21	Preprocessing in Python	Detailed notes 5
22	Section: feature generation (BoW, embeddings)	Detailed notes 6
23	BoW simple example	Detailed notes 6
24	Real term-document matrix	Detailed notes 6; Figures
25	BoW feature generation (4 vector types)	Detailed notes 6
26	TF-IDF	Detailed notes 6; Formulas
27	Feature generation in RapidMiner/Python	Detailed notes 6
28	Word embeddings	Detailed notes 6; Figures
29	Embedding methods, pretrained models	Detailed notes 6
30	Section: feature selection	Detailed notes 7

Previous: (none, first chunk) | Next: 01 Intro - Chunk 02 (s.31-48) | Up: 01 Intro - Index

Khaled Notes

Explorer

01 Intro - Chunk 01 (s.1-30)

01 Intro - Chunk 01 (s.1-30)

Overview

Detailed notes

1. Motivation and definition (s.1-4)

2. Text mining applications (s.5-10)

3. Search versus Discovery (s.10)

4. The Text Mining Process (s.11, s.30)

5. Text Preprocessing (s.12-21)

Stopword removal (s.15-16)

Stemming (s.17-19)

Tools (s.20-21)

6. Feature Generation (s.22-29)

Bag-of-Words (s.23-27)

Word Embeddings (s.28-29)

7. Feature Selection (s.30)

Figures & diagrams

Key terms

Formulas

Likely exam questions

Common confusions

Slide coverage

Graph View

Table of Contents

Backlinks