斯坦福 Speech and Language Processing - 2.4 Text normalization

Every NLP task requires text normalization:

1. Tokenzing (segmenting) words
2. Normalizing word formats
3. Segmenting sentences

2.4.1 Simple Tokenization in UNIX

Given a text file, output the word tokens and their frequencies

tr -sc ’A-Za-z’ ’\n’ < sh.txt
tr -sc ’A-Za-z’ ’\n’ < sh.txt | sort | uniq -c 
tr -sc ’A-Za-z’ ’\n’ < sh.txt | tr A-Z a-z | sort | uniq -c 
tr -sc ’A-Za-z’ ’\n’ < sh.txt | tr A-Z a-z | sort | uniq -c | sort -n -r 

2.4.2 Word Tokenization 分词

Tokenization in NLTK

中文一般不需要word tokenization

2.4.3 Byte Pair Encoding tokenization 字节对编码分词

• Subword tokenization (because tokens are often parts of words)
  Can include common morphemes like -est or -er.
  (A morpheme is the smallest meaning-bearing unit of a language：unlikeliest has morphemes un-, likely, and -est.)

• Three common algorithms:

1. Byte-Pair Encoding (BPE) (Sennrich et al., 2016)
2. unigram language modeling tokenization (Kudo, 2018)
3. WordPiece (Schuster and Nakajima, 2012)
   All have 2 parts:

• A token learner that takes a raw training corpus and induces a vocabulary (a set of tokens).
• A token segmenter that takes a raw test sentence and tokenizes it according to that vocabulary

BPE token learner algorithm

So first add a special end-of-word symbol ‘__’ before whitespace in training corpus

2.4.4 Word Normalization,Lemmatization and Stemming

Word Normalization 词标准化

Putting words/tokens in a standard format
例如：
U.S.A. or USA
uhhuh or uh-huh
Fed or fed
am, is be, are

Case folding 大写转小写

有些情况需要这样做，有些不需要

• For speech recognition and information retrieval, everything is mapped to lower case
• For sentiment analysis and text classification, IE, MT, case is helpful

Lemmatization 词形还原

Represent all words as their shared root, = dictionary headword form:

• am, are, is -> be
• car, cars, car’s, cars’ -> car

Stemming 词干提取

Reduce terms to stems, chopping off affixes crudely

Porter Stemmer

• Based on a series of rewrite rules run in series
  A cascade, in which output of each pass fed to next pass
  Some sample rules: