自然语言处理NLP Lecture Note for CS181 ShanghaiTech

说明：笔记旨在整理我校CS181课程的基本概念。由于授课及考试语言为英文，故英文出没可能。

目录

1 Formal grammers

2 regular grammars

3 dependency grammar

4 parsing

5 Learning grammars

5.1 Supervised methods

5.2 Unsupervised methods:

Reference

Appendix

---

1 Formal grammers

0.syntax: knowledge of the structural relationships between words

1.constituents: groups of words within sentences can be shown to act as single units.

2.grammer: the set of constituents and the rules that govern how they combine

3.CFGs(context-free grammers): aka: Phrase structure grammars

4. four components of a context-free grammar: a set of terminals \a set N of nonterminals(phrases)\a start symbol S\A set R of production rules(specifies how a nonterminal can produce a string of terminals and/or nonterminals)

5. grammatical structure(parse tree) of the string: ①starting from a string containing only the start symbol S② recursively applying rules to rewrite string ③ until the string contains only terminals

6.parsing: parsing is the process of taking a string and a grammar and returning one or more parse tree(s) for that string

7.probabilistic grammers: each rule is associated with a probaility. The probability of a parse tree is the product of the prob. of all the rules in that tree.

8. a sentence is ambiguous if it has more than one possible parse tree.

2 regular grammars

1. What is regular grammers? Only contain rules of form A->aB(word+parse) or A->a(word). Now, Probabilistic RG(regular grammar) = HMM(hidden markov models)

3 dependency grammar

1. A dependency grammar focuses on just binary relations among the words in a sentece while CFG focuses on consitituents.

2. A dependency parse is a tree where: ① the nodes are the words in a sentence. ② the links between the words represent their dependency relations.

3. advantage: deal well with free word order languages where the constituent structure is quite fulid. parsing is much faster than CFG-bases parsers. Dependency structure often captures the syntactic relations need by later applications(CFG capture it from tree anyway)

4. two modern approach to dependency parsing: ① optimization-based approaches. ② Shift-reduce approaches

5. DG vs. CFG

4 parsing

1.parsing with CFGs is the task of assigning proper parse trees to input strings

2. brute-force approach: enumerate all possible trees(high time cost)

3. DP(dynamic programming): a better way. CYK(Cock-Younger-Kasami Algorithm) is a bottom-up dynamic programming alogirhtm to solve sentence into CNF(Chomsky Normal Form).

4. CNF: only two types: A->B,C(nonterminal), A->w(terminal)

5. conversion to CNF: A->B, B->C turns to A->C/S->ABC turns to ->S->XC, X->AB

6. CYK algorithm

after filling table, recursively retrieve the constituents from the top(starting symbol) down.

7. to break tie(ambiguity when retrieve the constituents), we introduce probabilistic grammar, e.g: PCFG. TO find the parse tree with highest probability.

8.HMMs are special case of PCFGs, CYK on HMMs=Viterbi

5 Learning grammars

5.1 Supervised methods

1.treebank: a corpus(全集) in which each sentence has been (manually) paired with a parse tree.

2.key idea: max P(sentence, parse) by MLE

3.result: bad performance, may due to standard treebank nonterminals are not suffiicently informative

4. more notations

latent variable grammers: each nonterminal is split into a finite number of subtypes/ each subtype rule is with probability

discriminative parsing: we assume a weighted context-free grammar and maximize conditional likelihood P(parse | sentence).

5.2 Unsupervised methods:

two tasks: structure seach/ parameter learning

key idea: maximum (marginal) likelihood estimation. maximize P(sentence) with the parse tree marginalized with expecti-maximization.

 

Reference

1. Shanghaitech CS181 NLP.

Appendix

example of terminals, nonterminals and rules

1.nonterminals->nonterminals

2.nonterminals->terminals