CS224N NLP

附大佬的笔记：
github.com/LooperXX/LooperXX.github.io.git

Abbreviation

-	-
[ToL]	To learn
[ToLM]	To learn more
[ToLO]	To learn optionally
(0501)	05 min 01s
(h0501)	1 hour 05 min 01s
(hh0501)	2 hour 05 min 01s

Lecture 1 - Introduction and Word Vectors

NLP

Convert one-hot encoding to distributed representitions

Ont hot can’t represent the relation between word vectors,it is too big

Word2vec

Ignore the position of word of context

Use two vector in one word: centor word context word.

softmax function

Train the model: gradient descent

There is a term to calculate the gradient descent. (39:50-56:40)

result is :

ToL

Review derivation and the following especially.

Show some achievement with code(5640-h0516)

• We can do vector addition, subtraction, multiplication and division, etc.

QA

Why are there center word and context word(h0650)

To avoid one vector dot product himself in some situation???

Even synonyms can be merged into a vector(h1215)

Which is different from lee ,He says synonyms use different.

Lecture 2 Word Vectors,Word Senses,and Neural Classifiers

Bag models (0245)

The model makes the same predictions at each position.

Gradient descent (0600)

Not usually use because of the big calculation.

step size: not too big nor too small

stochastic gradient descent SGD TOBELM (0920)

Take part of the corpus

billion faster.

Maybe even get better result.

But it is stochastic, either you need sparse matrix update operations to only update certain rows of full embedding matrices U and V, or you need to keep around a hash for vectors.(1344)ToL

more details of word2vec(1400)

SG use center to predict context

SGNS negative sampling [ToBLO]

use logistic function instead of softmax and take sampling of corpus

CBOW opposite.

Why use two vectors(1500)

Sometime it will dot product with itself.

[ToL]

The first one is positive word and the last is negative word (2800)

negative word is being sampled cause the center word will turn up on other occasions, when it does, there will have other sampling, and it will learn step by step.

Why not capture co-occurrence counts directly?(2337)

SVD(3230) [ToL]

https://zhuanlan.zhihu.com/p/29846048

use svd to get lower dimensional representations for words

(3451)

Count based vs direct prediction

(3900)

Encoing meaning components in vector differences(3948)

This is to make addition subtraction available for word vectors.

GloVe (4313)

let dot product minus log of the co-occurrence

How to evaluate word vectors Intrinsic vs. extrinsic(4756)

Analogy evaluation and hyperparameters (intrinsic)(5515)

Word vector distances and their correlation with human judgements(5640)

Data shows that 300 dimensional word vector is good(5536)

The objective function for the GloVe model and What log-bilinear means(5739)

Word senses and word sense ambiguity(h0353)

One word different mean different vector.

then a word can be the sum of them all

It will work good but not bad (h1200)

the vector is so sparse that you can separate out different senses (h1402)

Lecture 3 Gradients by hand(matric calculus) and algorithmically(the backpropagation algorithm) all the math details of doing nerual net learning

Need to be learn again, it is not totally understanded.

Named Entity Recognition(0530)

Simple NER (0636)

How the sample model run (0836)

update equation(1220)

jacobian(1811)

Chain Rule(2015)

do one example step (2650)

hadamard product ToL

Reusing Computation(3402)

ds/dw

Forward and backward propagation(5000)

An example(5507)

a = x+y

b = max(y,z)

f = ab

Compute all gradients at once (h0005)

Back-prop in general computation graph(h0800)[ToL]

Automatic Differentiation(h1346)

Many tools can calculate automaticly.

Manual Gradient checking : Numeric Gradient(h1900)

Lecture 4 Dependency Parsing

Two views of linguistic structure

Constituency = phrase structure grammar = context-free grammars(CFGs)(0331)

Phrase structure organizes words into nested constituents

Dependency structure(1449)

Dependency structure shows which words depend on (modify, attach to,or are arguments of)

Why do we need sentence structure?(2205)

Can not express meaning by just one word.

Prepositional phrase attachment ambiguity.(2422)

There is some sentence to show it:

San Jose cops kill man with knife

Scientists count whales from space

The board approved [its acquisition] [by Royal Trustco Ltd.] [of Toronto] [for $27 a share] [at its monthly meeting].

Coordination scope ambiguity(3614)

**Shuttle veteran and longtime NASA executive Fred Gregory appointed to board **

Doctor: No heart, cognitive issues

Adjectival/Adverbial Modifier Ambiguity(3755)

Students get [first hand job] experience Students get first [hand job] experience

Verb Phrase(VP) attachment ambiguity(4404)

Mutilated body washes up on Rio beach to be used for Olympics beach volleyball.

Dependency Grammar and Dependency structure(4355)

Will add a fake ROOT for handy

Dependency Grammar history(4742)

The rise of annotated data Universal Dependency tree(5100)

Tree bank(5400)

Its too slow to write a grammar by hand but its still worth,cause it can used in another place but not only nlp .

how to build parser with dependency(5738)

Dependency Parsing

Projectivity(h0416)

Methods of Dependency Parsing(h0521)

Greedy transition-based parsing(h0621)

Basic transition-based dependency parser (h0808)

[root] I ate fish

[root I ate] fish

[root ate] fish

[root ate fish]

[root ate]

[root]

MaltParser(h1351)[ToL]

Evaluation of Dependency Parsing (h1845)[ToL]

Lecture-5 Languages models and Recurrent Neural Networks(RNNs)

A neural dependency parser(0624)

Distributed Representations(0945)

Deep Learning Classifier are non-linear classifiers(1210)

Deep Learning Classifier’s non-linear classifiers:

Simple feed-forward neural network multi-class classifier (1621)

Neural Dependency Parser Model Architecture(1730)

Graph-based dependency parsers (2044)

Regularization && Overfitting (2529)

Dropout (3100)[ToL]

Vectorization(3333)

Non-linearities (4000)

Parameter Initialization (4357)

Optimizers(4617)

Learning Rates(4810)

It can be slow as the learning go on.

Language Modeling (5036)

n-gram Language Models(5356)

Sparsity Problems (5922)

Many situation didn’t occur so it will be zero

Storage Problems(h0117)

How to build a neural language model(h0609)

A fixed-window neural Language Model(h1100)

Recurrent Neural Network (RNN)(h1250)

x1 -> y1

Wx1 x2 -> y1

A Simple RNN Language Model(h1430)

Lecture 6 Simple and LSTM Recurrent Neural Networks.

The Simple RNN Language Model (0310)

Training an RNN Language Model (0818)

RNN takes more time.

Teacher Forcing

penalize when dont take its advise

But how do we get the answer?

Evaluating Language Models (2447)[ToL]

Language Model is a system that predicts the next word(3130)

Other use of RNN(3229)

Tag for word

Used for classification(3420)

Used to Language encoder module (3500)

Used to generate text (3600)

Problems with Vanishing and Exploding Gradients(3750)[IMPORTANT]

[ToL]

Why This is a problem (4400)

We can give him a limit.

Long Short Term Memory RNNS(LSTMS)(5000)[ToL]