基本循环神经网络误差反向传播公式

1.前向传播公式

$$z^t=U\cdot x^t+W\cdot h^{t-1}+B$$$$h^t=f\left(z^t\right)$$$$o^t=V\cdot h^t+C$$$$y^t=g\left(o^t\right)$$$$L=\sum _{t=1}^T{l}^t$$$$l^t=e\left(y^t\right)$$

2. $\frac{\partial L}{\partial V}$与$\frac{\partial L}{\partial C}$

$$dL=tr\left(\sum _{t=1}^T{\frac{\partial l^t}{\partial y^t}}^T\cdot d{y}^t\right)$$$$=tr\left(\sum _{t=1}^T{\frac{\partial l^t}{\partial y^t}}^T\cdot \left(g^{{}^{\prime }}\left(o^t\right)\odot \left(dV\cdot h^t+dC\right)\right)\right)$$$$=tr\left(\sum _{t=1}^T\left({\frac{\partial l^t}{\partial y^t}}^T\odot {\left(g^{{}^{\prime }}\left(o^t\right)\right)}^T\right)\cdot \left(dV\cdot h^t+dC\right)\right)$$$$=tr\left(\sum _{t=1}^T{h}^t\cdot \left({\frac{\partial l^t}{\partial y^t}}^T\odot {\left(g^{{}^{\prime }}\left(o^t\right)\right)}^T\right)\cdot dV+\sum _{t=1}^T\left({\frac{\partial l^t}{\partial y^t}}^T\odot {\left(g^{{}^{\prime }}\left(o^t\right)\right)}^T\right)\cdot dC\right)$$

$$\frac{\partial L}{\partial V}=\sum _{t=1}^T\left(\frac{\partial l^t}{\partial y^t}\odot g^{{}^{\prime }}\left(o^t\right)\right)\cdot {\left(h^t\right)}^T$$$$\frac{\partial L}{\partial C}=\sum _{t=1}^T\frac{\partial l^t}{\partial y^t}\odot g^{{}^{\prime }}\left(o^t\right)$$

3. $\frac{\partial L}{\partial U}$,$\frac{\partial L}{\partial W}$,$\frac{\partial L}{\partial B}$

$$dL=tr\left(\sum _{t=1}^T{\frac{\partial L}{\partial z^t}}^T\cdot d{z}^t\right)$$$$=tr\left(\sum _{t=1}^T{\frac{\partial L}{\partial z^t}}^T\cdot \left(dU\cdot x^t+dW\cdot h^{t-1}+dB\right)\right)$$$$=tr\left(\sum _{t=1}^T{x}^t\cdot {\frac{\partial L}{\partial z^t}}^T\cdot dU+\sum _{t=1}^T{h}^{t-1}\cdot {\frac{\partial L}{\partial z^t}}^T\cdot dW+\sum _{t=1}^T{\frac{\partial L}{\partial z^t}}^T\cdot dB\right)$$

$$\frac{\partial L}{\partial U}=\sum _{t=1}^T\frac{\partial L}{\partial z^t}\cdot {\left(x^t\right)}^T$$$$\frac{\partial L}{\partial W}=\sum _{t=1}^T\frac{\partial L}{\partial z^t}\cdot {\left(h^{t-1}\right)}^T$$$$\frac{\partial L}{\partial B}=\sum _{t=1}^T\frac{\partial L}{\partial z^t}$$

4. $\frac{\partial L}{\partial x^t}$

$$dL=tr\left({\frac{\partial L}{\partial z^t}}^T\cdot d{z}^t\right)$$$$=tr\left({\frac{\partial L}{\partial z^t}}^T\cdot U\cdot d{x}^t\right)$$

$$\frac{\partial L}{\partial x^t}=U^T\cdot \frac{\partial L}{\partial z^t}$$

5. $\frac{\partial L}{\partial z^t}$

5.1 $\frac{\partial l^t}{\partial z^t}$

$$d{l}^t=tr\left({\frac{\partial l^t}{\partial y^t}}^T\cdot d{y}^t\right)$$$$=tr\left({\frac{\partial l^t}{\partial y^t}}^T\cdot \left(g^{{}^{\prime }}\left(o^t\right)\odot d{o}^t\right)\right)$$$$=tr\left(\left({\frac{\partial l^t}{\partial y^t}}^T\odot {\left(g^{{}^{\prime }}\left(o^t\right)\right)}^T\right)\cdot V\cdot d{h}^t\right)$$$$=tr\left(\left({\frac{\partial l^t}{\partial y^t}}^T\odot {\left(g^{{}^{\prime }}\left(o^t\right)\right)}^T\right)\cdot V\cdot \left(f^{{}^{\prime }}\left(z^t\right)\odot d{z}^t\right)\right)$$$$=tr\left\{\left[\left(\left({\frac{\partial l^t}{\partial y^t}}^T\odot {\left(g^{{}^{\prime }}\left(o^t\right)\right)}^T\right)\cdot V\right)\odot {\left(f^{{}^{\prime }}\left(z^t\right)\right)}^T\right]\cdot d{z}^t\right\}$$

$$\frac{\partial l^t}{\partial z^t}=\left(V^T\cdot \left(\frac{\partial l^t}{\partial y^t}\odot g^{{}^{\prime }}\left(o^t\right)\right)\right)\odot f^{{}^{\prime }}\left(z^t\right)$$

5.2 $\frac{\partial l^k}{\partial z^t},\left(k\ge t+1\right)$

$$d{l}^k=tr\left({\frac{\partial l^k}{\partial z^{t+1}}}^T\cdot d{z}^{t+1}\right)$$$$=tr\left({\frac{\partial l^k}{\partial z^{t+1}}}^T\cdot W\cdot \left(f^{{}^{\prime }}\left(z^t\right)\odot d{z}^t\right)\right)$$$$=tr\left\{\left(\left({\frac{\partial l^k}{\partial z^{t+1}}}^T\cdot W\right)\odot {\left(f^{{}^{\prime }}\left(z^t\right)\right)}^T\right)\cdot d{z}^t\right\}$$

$$\frac{\partial l^k}{\partial z^t}=\left(W^T\cdot \frac{\partial l^k}{\partial z^{t+1}}\right)\odot f^{{}^{\prime }}\left(z^t\right)$$

5.3 $\frac{\partial L}{\partial z^t}$递推公式

$$\frac{\partial L}{\partial z^t}=\frac{\partial l^t}{\partial z^t}+\sum _{k=t+1}^T\frac{\partial l^k}{\partial z^t}$$$$=\frac{\partial l^t}{\partial z^t}+\left(W^T\cdot \sum _{k=t+1}^T\frac{\partial l^k}{\partial z^{t+1}}\right)\odot f^{{}^{\prime }}\left(z^t\right)$$$$=\left(V^T\cdot \left(\frac{\partial l^t}{\partial y^t}\odot g^{{}^{\prime }}\left(o^t\right)\right)\right)\odot f^{{}^{\prime }}\left(z^t\right)+\left(W^T\cdot \frac{\partial L}{\partial z^{t+1}}\right)\odot f^{{}^{\prime }}\left(z^t\right)$$

6.补充说明

只有比$t$时刻更晚的$l^t,l^{t+1},\dots ,l^T$才会与$z^t$有关，所以当$k<t$时，$$\frac{\partial l^k}{\partial z^t}=0$$$$及:\frac{\partial L}{\partial z^t}=\sum _{k=1}^T\frac{\partial l^k}{\partial z^t}=\sum _{k=t}^T\frac{\partial l^k}{\partial z^t}$$