目录
Every NLP task requires text normalization:
- Tokenzing (segmenting) words
- Normalizing word formats
- Segmenting sentences
2.4.1 Simple Tokenization in UNIX
Given a text file, output the word tokens and their frequencies
tr -sc ’A-Za-z’ ’\n’ < sh.txt
tr -sc ’A-Za-z’ ’\n’ < sh.txt | sort | uniq -c
tr -sc ’A-Za-z’ ’\n’ < sh.txt | tr A-Z a-z | sort | uniq -c
tr -sc ’A-Za-z’ ’\n’ < sh.txt | tr A-Z a-z | sort | uniq -c | sort -n -r
2.4.2 Word Tokenization 分词
Tokenization in NLTK
中文一般不需要word tokenization
2.4.3 Byte Pair Encoding tokenization 字节对编码分词
-
Subword tokenization (because tokens are often parts of words)
Can include common morphemes like -est or -er.
(A morpheme is the smallest meaning-bearing unit of a language:unlikeliest has morphemes un-, likely, and -est.) -
Three common algorithms:
- Byte-Pair Encoding (BPE) (Sennrich et al., 2016)
- unigram language modeling tokenization (Kudo, 2018)
- WordPiece (Schuster and Nakajima, 2012)
All have 2 parts:
- A token learner that takes a raw training corpus and induces a vocabulary (a set of tokens).
- A token segmenter that takes a raw test sentence and tokenizes it according to that vocabulary
BPE token learner algorithm
So first add a special end-of-word symbol ‘__’ before whitespace in training corpus
2.4.4 Word Normalization,Lemmatization and Stemming
Word Normalization 词标准化
Putting words/tokens in a standard format
例如:
U.S.A. or USA
uhhuh or uh-huh
Fed or fed
am, is be, are
Case folding 大写转小写
有些情况需要这样做,有些不需要
- For speech recognition and information retrieval, everything is mapped to lower case
- For sentiment analysis and text classification, IE, MT, case is helpful
Lemmatization 词形还原
Represent all words as their shared root, = dictionary headword form:
- am, are, is -> be
- car, cars, car’s, cars’ -> car
Stemming 词干提取
Reduce terms to stems, chopping off affixes crudely
Porter Stemmer
- Based on a series of rewrite rules run in series
A cascade, in which output of each pass fed to next pass
Some sample rules: