Word Vector Representation

1. SVD Based Methods

   1.1 Word-Document Matrix

   1.2 Window based CO-occurrence Matrix

   In this method we count the number of times each word appears inside a window of a particular size around the word of interest. We calculate this count for all the words in corpus.

   1.3 Advantages:Both of these methods give us word vectors that are more than sufficient to encode semantic and syntacic

   1.4 Shortcomming:

   $\bigstar$ The dimensions of the matrix change very often(new words are added very frequently and corpus changes in size)

   $\bigstar$ The matrix is extremely sparse since most words do not co-occur

   $\bigstar$ The matrix is very hign dimensional in general

   $\bigstar$ Quadratic cost to train(perform SVD)

---

1. Iteration Based Methods

   2.1 CBOW Model

   $\blacktriangle$ key idea: Predicting a center word from the surrounding context

   $\blacktriangle$ unkonwns: Two matrics,$V \in R^{n\times \vert{V}\vert}$ and $U\in R^{\vert{V}\vert \times n}$

   $\blacktriangle$ Notation for CBOW Model:

   • $w_i$:Word i from vocabulary V

   • $V \in R^{n \times \vert{V}\vert}$:Input word matrix

   • $v_i$:the input vector representation of word $w_i$

   • $U \in R^{n \times \vert{V}\vert}$:Output word matrix

   • $u_i$:the output vector representation of word $w_i$

   $\blacktriangle$ Steps:

   • We generate our one hot word vector($x^{(c-m)}$,…,$x^{(c-1)}$,$x^{(c+1)}$,…,$x^{(c+m)}$) for the input context of size m.

   • We get our embedded word vectors for the context ($V_{c-m}=Vx^{(c-m)}$,$V_{c-m+1}=Vx^{(c-m+1)}$,…$V_{c+m}=Vx^{(c+m)}$)

   • Average these vectors to get $\hat{v}=\frac{v_{c-m}+v_{c-m+1}+...+v_{c+m}}{2m}$

   • Generate a score vector $z=U\hat{v}$

   • Turn the scores into probabilities $\hat{y}=sofrmax(z)$

   • We desire our probabilities generated,$\hat{y}$,to match the true probabilities,y,which also happens to be the one hot vector of the actual word
     

   2.2 Skip-Gram Model

   $\blacktriangle$key idea: Predicting surrounding context words given a center word

   $\blacktriangle$steps:

   • We generate our one hot input vector x

   • We get our embedded word vectors for the context $v_c=Vx$

   • Since there is no averaging,just set $\hat{v}=v_c$

   • Generate 2m score vectors,$u_{c-m},...,u_{c-1},u_{c+1},...,u_{c+m}$ using $u=Uv_c$

   • Turn each of the scores into probabilitiesm, $y=softmax(u)$

   • We desire our probability vector generated to match the true probabilities which is $y^{c-m},...,y^{c-1},y^{c+1},...,y^{c+m}$,the one hot vecotrs of the actual output