02 Pre-processing - Mind Map

Up: 02 Pre-processing - Index | 02 Pre-processing - Cheat Sheet

mindmap
  root((Pre-processing))
    Features
      Feature extractor
      Linear model
    Text normalization
      Tokenization
      Tokens vs types
      Zipf law
      Stopwords
      Case folding
      Stemming
      Lemmatization
    Linguistic analysis
      Sentence boundaries
      POS tagging
      WSD and Lesk
      Parsing and UD
      WordNet
    Bag of words
      Document-term matrix
      tf-idf
      Multiword features
      CSR sparse storage
    Reduction
      Dictionary reduction
      Entropy and IG
      Feature selection
    Similarity
      Cosine similarity
    LSA
      SVD
      Truncated SVD
      Synonymy
      NMF and others

flowchart LR
  A[Raw text] --> B[Sentence split]
  B --> C[Tokenize]
  C --> D[Normalize: case, stop, stem/lemma]
  D --> E[Bag of words matrix]
  E --> F{Too big?}
  F -->|keep subset| G[Feature selection: IG]
  F -->|combine| H[SVD / LSA]
  G --> I[Model]
  H --> I
  E --> J[Cosine similarity]