CS224N notes_chapter5_Backpropagation

第五讲 Backpropagation

From one-layer NN to multi layer NN

2 layer case.
$$\begin{array}{rl}x=& z^{(1)}=a^{(1)}\\ z^{(2)}=& W^{(1)}x+b^{(1)}\\ a^{(2)}=& f(z^{(2)})\\ z^{(3)}=& W^{(2)}{a}^{(2)}+b^{(2)}\\ a^{(3)}=& f(z^{(3)})\\ s=& U^T{a}^{(3)}\end{array}$$
for $W^{(2)}$,
$$\frac{\partial s}{\partial W_{ij}^{(2)}}={\delta }_i^{(3)}{a}_j^{(2)}$$
In matrix notation
$$\begin{gathered} \frac{\partial s}{\partial W^{(2)}}={\delta }^{(3)}{a}^{(2{)}^T} \\ {\delta }^{(3)}=U\odot f^{\prime }(z^{(3)}) \end{gathered}$$
Then, we need to calculate $\frac{\partial s}{\partial W^{(1)}}$
$$\begin{array}{rl}s=& U^{T}f(W^{(2)}f(W^{(1)}x+b^{(1)})+b^{(2)})\\ =& U^{T}f(W^{(2)}f([...;W_{ij}^{(1)}{x}_j+C+b_j^{(1)};...])+b^{(2)})\\ =& U^{T}f(W^{(2)}\left[\begin{array}{c}...\\ f(W_{ij}^{(1)}{x}_j+C+b_j^{(1)});\\ ...\end{array}\right]+b^{(2)})\\ =& U^{T}f(\left[\begin{array}{c}{W}_{1i}^{(2)}f(W_{ij}^{(1)}{x}_j+C+b_j^{(1)})+C\\ W_{2i}^{(2)}f(W_{ij}^{(1)}{x}_j+C+b_j^{(1)})+C\\ ...\\ W_{ni}^{(2)}f(W_{ij}^{(1)}{x}_j+C+b_j^{(1)})+C\end{array}\right]+b^{(2)})\\ =& U^{T}f(\left[\begin{array}{c}{W}_{1i}^{(2)}f(W_{ij}^{(1)}{x}_j+C+b_j^{(1)})+C+b_1^{(2)}\\ W_{2i}^{(2)}f(W_{ij}^{(1)}{x}_j+C+b_j^{(1)})+C+b_2^{(2)}\\ ...\\ W_{ni}^{(2)}f(W_{ij}^{(1)}{x}_j+C+b_j^{(1)})+C+b_n^{(2)}\end{array}\right])\\ =& U^T\left[\begin{array}{c}f(W_{1i}^{(2)}f(W_{ij}^{(1)}{x}_j+C+b_j^{(1)})+C+b_1^{(2)})\\ f(W_{2i}^{(2)}f(W_{ij}^{(1)}{x}_j+C+b_j^{(1)})+C+b_2^{(2)})\\ ...\\ f(W_{ni}^{(2)}f(W_{ij}^{(1)}{x}_j+C+b_j^{(1)})+C+b_n^{(2)})\end{array}\right]\\ =& \sum _k{U}_{k}f(W_{ki}^{(2)}f(W_{ij}^{(1)}{x}_j+C+b_j^{(1)})+C+b_k^{(2)})\end{array}$$
Thus,
$$\begin{array}{rl}\frac{\partial s}{\partial W_{ij}^{(1)}}=& \sum _k{U}_k{f}_k^{\prime }(z^{(3)})f_k^{\prime }(z^{(2)})W_{ki}^{(2)}{x}_j\\ =& W_{\cdot i}^{(2{)}^T}{f}^{\prime }(z^{(2)})\odot U\odot f^{\prime }(z^{(3)})x_j\\ \frac{\partial s}{\partial W_{i\cdot }^{(1)}}=& W_{\cdot i}^{(2{)}^T}{f}^{\prime }(z^{(2)})\odot U\odot f^{\prime }(z^{(3)})x^T\\ \frac{\partial s}{\partial W^{(1)}}=& W^{(2{)}^T}{f}^{\prime }(z^{(2)})\odot U\odot f^{\prime }(z^{(3)})x^T\\ =& W^{(2{)}^T}{\delta }^{(3)}\odot f^{\prime }(z^{(2)})x^T\end{array}$$
#我也不知道我自己推的对不对，反正和课件里的结果对上号了，欢迎读者批评指正。
$$\begin{array}{rl}{\delta }^{(l)}=& ((W^{(l)}{)}^T{\delta }^{(l+1)})\odot f^{\prime }(z^{(l)})\\ \frac{\partial }{\partial W^{(l)}}{E}_R=& {\delta }^{(l+1)}(a^{(l+1)})(a^{(l)}{)}^T+\lambda W^{(l)}\end{array}$$

BP

$$f(x,y,z)=(x+y)z$$
def:
$$\begin{array}{rlr}q=x+y& & \frac{\partial q}{\partial x}=1,\frac{\partial q}{\partial y}=1\\ f=qz& & \frac{\partial f}{\partial q}=z,\frac{\partial f}{\partial z}=q\end{array}$$
then we could use chain rule.

sequenceDiagram
x->>q:
y->>q:
q->>z

Recursively apply chain rule through each node.

Another 3 different descriptions and viewpoints of bp

• Functions as Circuits
• High-level flowgraph; move back through the graph.
• Delta error signals in real NN

Papaer Reading

FastText:

• average word vector.
• hierarchical softmax
  • we construct a Huffuman Tree based on word frequency. Each non-leaf node is a two-class classifier. And each leaf denotes a word.