Concept - Sentence Boundary Detection

Definition (s.37-38): deciding which periods followed by whitespace are sentence delimiters. ? and ! are assumed unambiguous; . is ambiguous (sentence end, abbreviations like Inc./Dr., numbers like .02%/4.3).

Why (s.37): sophisticated parsing, sentence-level sentiment analysis and many information extraction algorithms need whole sentences as input.

Approach (s.38): binary classifier on each ”.” -> EndOfSentence / NotEndOfSentence; implemented with hand-written rules, regular expressions or machine learning.

Decision tree (s.39): blank lines after? -> EOS; final punctuation ? ! : -> EOS; final punctuation is period? no -> not EOS; abbreviation like “etc”? yes -> not EOS, no -> EOS.

Hand-written algorithm (s.41): (1) replace identifiable non-EOS periods with another character; (2) apply rules and mark EOS periods; (3) retransform step-1 characters to periods; (4) EOS periods are now identified. Examples not EOS: Ph.D., www.google.com, i.e.

Decision trees (s.40): just if-then-else; hand-building only for simple features; usually learned from a training corpus; features can be used by SVM, NN, logistic regression.

flowchart TD
    A{Blank lines after?} -- yes --> E[EOS]
    A -- no --> B{Final punct ? ! :} 
    B -- yes --> E
    B -- no --> C{Period?}
    C -- no --> N[Not EOS]
    C -- yes --> D{Abbreviation?}
    D -- yes --> N
    D -- no --> E

Related

Concept - Tokenization | Concept - Text Preprocessing | Concept - Parsing

Source: 02 Pre-processing - Chunk 02 (s.31-60)