03 Text Classification - Index

Quick links: 03 Text Classification - Cheat Sheet | 03 Text Classification - Mind Map | Canvas: 03 Text Classification - Canvas.canvas

Chunk notes (in lecture order)

1. 03 Text Classification - Chunk 01 (s.1-30): recap of document representation; probability pre-study (conditional probability, density estimation, argmax, MLE); Bayes rule, MAP vs MLE, conditional independence; Naive Bayes for text, mega-document estimation, new-word problem, additive smoothing.
2. 03 Text Classification - Chunk 02 (s.31-60): out-of-vocabulary words, scikit-learn MultinomialNB; start of the normal-vs-spam worked example: word histograms to word likelihoods; the “Lunch” zero in spam.
3. 03 Text Classification - Chunk 03 (s.61-90): spam example continued: priors p(N)=0.67, p(S)=0.33; scores for “Dear Friend” (0.09 normal vs 0.01 spam).
4. 03 Text Classification - Chunk 04 (s.91-120): “Dear Friend” classified normal; “Lunch Money Money Money Money” gets spam score 0 (zero-probability problem); add a pseudo-count (Laplace smoothing).
5. 03 Text Classification - Chunk 05 (s.121-150): smoothing fixes the example; why Naive Bayes is “naive” (bag of words) and still works (high bias, low variance); Multinomial vs Bernoulli NB.
6. 03 Text Classification - Chunk 06 (s.151-180): MNB vs BNB worked comparison; posterior normalization; start of Gaussian Naive Bayes (Troll 2 example).
7. 03 Text Classification - Chunk 07 (s.181-210): Gaussian NB worked example: priors, likelihoods from Gaussians, log trick against underflow.
8. 03 Text Classification - Chunk 08 (s.211-240): Gaussian NB log scores (Loves -124 vs Does not love about -48); classification result.
9. 03 Text Classification - Chunk 09 (s.241-270): candy decides; cross validation for feature choice; linear regression review and why it is a poor classifier; sigmoid, logistic regression, cross-entropy loss, gradient descent, decision boundary.
10. 03 Text Classification - Chunk 10 (s.271-300): logistic regression with BoW; confusion matrix, precision/recall, F1, ROC/AUC; multiclass (one-vs-all, softmax); generative vs discriminative; decision tree terminology.
11. 03 Text Classification - Chunk 11 (s.301-330): decision trees: ID3, entropy/information gain, continuous attributes, gain ratio, Gini; pros and cons; start of overfitting.
12. 03 Text Classification - Chunk 12 (s.331-360): generalization, over/underfitting, pruning; ensemble learning, weak/strong and stable/unstable learners, bias-variance; bagging, random forest, boosting, stacking.
13. 03 Text Classification - Chunk 13 (s.361-381): Netflix Prize, No Free Lunch; parting remarks: sentiment analysis, word sense disambiguation; getting labels and inter-annotator agreement.

Concept notes by topic

Probability foundations

Concept - Probability Basics, Concept - Conditional Probability, Concept - Conditional Independence, Concept - Density Estimation, Concept - Argmax, Concept - Likelihood, Concept - Maximum Likelihood Estimation, Concept - Prior Probability, Concept - Posterior Probability, Concept - Bayes Rule, Concept - Maximum A Posteriori, Concept - Curse of Dimensionality

Naive Bayes

Concept - Naive Bayes, Concept - Naive Bayes Classifier, Concept - Multinomial Naive Bayes, Concept - Bernoulli Naive Bayes, Concept - Gaussian Naive Bayes, Concept - Word Likelihood, Concept - Out of Vocabulary (OOV), Concept - Zero Probability Problem, Concept - Additive Smoothing, Concept - Laplace Smoothing, Concept - Posterior Normalization, Concept - Underflow

Linear and probabilistic classifiers

Concept - Linear Regression, Concept - Sigmoid Function, Concept - Logistic Regression, Concept - Cross-Entropy Loss, Concept - Gradient Descent, Concept - Decision Boundary, Concept - One-vs-All Classification, Concept - Softmax Classifier, Concept - Generative vs Discriminative Models

Evaluation

Concept - Confusion Matrix, Concept - Precision and Recall, Concept - F1 Score, Concept - ROC Curve and AUC, Concept - Cross Validation

Decision trees

Concept - Decision Tree, Concept - ID3, Concept - Entropy, Concept - Information Gain, Concept - Gain Ratio, Concept - Gini Impurity, Concept - Splitting Continuous Attributes, Concept - Decision Tree Pruning

Generalization and ensembles

Concept - Generalization, Concept - Model Complexity, Concept - Overfitting, Concept - Underfitting, Concept - Bias and Variance, Concept - Bias-Variance Tradeoff, Concept - Ensemble Learning, Concept - Weak and Strong Learners, Concept - Stable and Unstable Learners, Concept - Bagging, Concept - Bootstrapping, Concept - Majority Voting, Concept - Random Forest, Concept - Boosting, Concept - Stacking, Concept - Netflix Prize, Concept - No Free Lunch Theorem

Applications and labels

Concept - Sentiment Analysis Related Tasks, Concept - Word Sense Disambiguation, Concept - WordNet and Synsets, Concept - Getting Labels, Concept - Inter-Annotator Agreement

Earlier-deck concepts referenced

Concept - TF-IDF, Concept - Bag of Words, Concept - Latent Semantic Analysis (LSA)