Lecture 2 - Simple Word Vector representations: word2vec, GloVe

How to represent meaning in a computer?

Discrete Representation：
One-Hot Representation

But the one-hot representation has a problem: hard to compute similarity.

Distributional Representation：
Full Document & Window Based
Full document, like word-document coocurrence matrix -> LDA -> suitable for text classification.
Window based, like the following:

But the window based method also has a problem: the matrix dimension is too high.

Solution: SVD, 什么是奇异值分解

But SVG has to cost much time!
So we think about obtaining low dimensional vectors directly.

Warm Up: Gradient Descent