LSTM模型与前向反向传播算法笔记

---

这篇文章是看了刘建平老师的LSTM模型与前向反向传播算法后的笔记，同时参考这两篇文章，包括一些公式推导，都是自己的理解，如有错误，欢迎指出。

---

LSTM前向传播算法

这边直接给出前向传播过程中的公式计算，具体的可以参考上面文章：

其中，
$$\begin{gathered} {\mathbf{x}}^{(t)}-n\times 1,{\mathbf{h}}^{(t)}-m\times 1 \\ \mathbf{V}-l\times m,\mathbf{c}-l\times 1,{\hat{\mathbf{y}}}^{(t)}-l\times 1 \\ {\mathbf{o}}^{(t)},{\mathbf{f}}^{(t)},{\mathbf{i}}^{(t)},{\mathbf{a}}^{(t)},{\mathbf{C}}^{(t)}-m\times 1 \\ {\mathbf{W}}_f,{\mathbf{W}}_i,{\mathbf{W}}_a,{\mathbf{W}}_o-m\times m \\ {\mathbf{U}}_f,{\mathbf{U}}_i,{\mathbf{U}}_a,{\mathbf{U}}_o-m\times n \\ {\mathbf{b}}_f,{\mathbf{b}}_i,{\mathbf{b}}_a,{\mathbf{b}}_o-m\times 1 \end{gathered}$$
$L(t)$分为两部分，一部分为$l(t)$，另一部分为$t$时刻之后的损失$L(t+1)$：
$$L(t)=\{\begin{array}{l}l(t)+L(t+1),t<\tau \\ l(t),t=\tau \end{array}$$

所以，当$t=\tau$时，我们有${\mathbf{\delta }}_h^{(\tau )}=\frac{\partial L}{\partial {\mathbf{h}}^{(\tau )}}={\mathbf{V}}^T({\hat{\mathbf{y}}}^{(\tau )}-{\mathbf{y}}^{(\tau )})$，具体的推导可以参考RNN，此外，
$$\begin{array}{rl}dL& ={\left(\frac{\partial L}{\partial {\mathbf{h}}^{(\tau )}}\right)}^{T}d{\mathbf{h}}^{(\tau )}\\ & ={\left({\mathbf{\delta }}_h^{(\tau )}\right)}^{T}d\left({\mathbf{o}}^{(\tau )}\odot tanh({\mathbf{C}}^{(\tau )})\right)\\ & =tr\left({\left({\mathbf{\delta }}_h^{(\tau )}\right)}^T\left({\mathbf{o}}^{(\tau )}\odot dtanh({\mathbf{C}}^{(\tau )})\right)\right)\\ & =tr\left({\left({\mathbf{\delta }}_h^{(\tau )}\odot {\mathbf{o}}^{(\tau )}\right)}^{T}dtanh({\mathbf{C}}^{(\tau )})\right)\\ & =tr\left({\left({\mathbf{\delta }}_h^{(\tau )}\odot {\mathbf{o}}^{(\tau )}\right)}^T\left(1-tan{h}^2({\mathbf{C}}^{(\tau )})\right)\odot d{\mathbf{C}}^{(\tau )}\right)\\ & =tr\left({\left[{\mathbf{\delta }}_h^{(\tau )}\odot {\mathbf{o}}^{(\tau )}\odot \left(1-tan{h}^2({\mathbf{C}}^{(\tau )})\right)\right]}^{T}d{\mathbf{C}}^{(\tau )}\right)\end{array}$$
所以，${\mathbf{\delta }}_C^{(\tau )}=\frac{\partial L}{\partial {\mathbf{C}}^{(\tau )}}={\mathbf{\delta }}_h^{(\tau )}\odot {\mathbf{o}}^{(\tau )}\odot \left(1-tan{h}^2({\mathbf{C}}^{(\tau )})\right)$。
接下来由$t+1$项往前推导：
$$\begin{array}{rl}dL& =dl(t)+dL(t+1)\\ & ={\left(\frac{\partial l(t)}{\partial {\mathbf{h}}^{(t)}}\right)}^{T}d{\mathbf{h}}^{(t)}+{\left(\frac{\partial L(t+1)}{\partial {\mathbf{h}}^{(t+1)}}\right)}^{T}d{\mathbf{h}}^{(t+1)}\\ & ={\left(\frac{\partial l(t)}{\partial {\mathbf{h}}^{(t)}}\right)}^{T}d{\mathbf{h}}^{(t)}+{\left(\frac{\partial L(t+1)}{\partial {\mathbf{h}}^{(t+1)}}\right)}^T{\left(\frac{\partial {\mathbf{h}}^{(t+1)}}{\partial {\mathbf{h}}^{(t)}}\right)}^{T}d{\mathbf{h}}^{(t)}\\ 所以：& \\ {\mathbf{\delta }}_h^{(t)}& =\frac{\partial L}{\partial {\mathbf{h}}^{(t)}}\\ & =\frac{\partial l(t)}{\partial {\mathbf{h}}^{(t)}}+\frac{\partial {\mathbf{h}}^{(t+1)}}{\partial {\mathbf{h}}^{(t)}}{\mathbf{\delta }}_h^{(t+1)}\\ 其中，\frac{\partial {\mathbf{h}}^{(t+1)}}{\partial {\mathbf{h}}^{(t)}}如下得出：& \\ d{\mathbf{h}}^{(t+1)}& =d{\mathbf{o}}^{(t+1)}\odot tanh({\mathbf{C}}^{(t+1)})\\ & =tanh({\mathbf{C}}^{(t+1)})\odot d{\mathbf{o}}^{(t+1)}+{\mathbf{o}}^{(t+1)}\odot dtanh({\mathbf{C}}^{(t+1)})\\ & =tanh({\mathbf{C}}^{(t+1)})\odot {\mathbf{o}}^{(t+1)}\odot (1-{\mathbf{o}}^{(t+1)})\odot d{\mathbf{W}}_o{\mathbf{h}}^{(t)}\\ & +{\mathbf{o}}^{(t+1)}\odot \left(1-tan{h}^2({\mathbf{C}}^{(t+1)})\right)\odot d{\mathbf{C}}^{(t+1)}\\ & =diag\left[tanh({\mathbf{C}}^{(t+1)})\odot {\mathbf{o}}^{(t+1)}\odot (1-{\mathbf{o}}^{(t+1)})\right]{\mathbf{W}}_{o}d{\mathbf{h}}^{(t)}\\ & +diag\left[{\mathbf{o}}^{(t+1)}\odot \left(1-tan{h}^2({\mathbf{C}}^{(t+1)})\right)\right]d{\mathbf{C}}^{(t+1)}\\ {\mathbf{C}}^{(t+1)}& ={\mathbf{C}}^{(t)}\odot {\mathbf{f}}^{(t+1)}+{\mathbf{i}}^{(t+1)}\odot {\mathbf{a}}^{(t+1)}\\ d{\mathbf{C}}^{(t+1)}& ={\mathbf{f}}^{(t+1)}\odot d{\mathbf{C}}^{(t)}+{\mathbf{C}}^{(t)}\odot d{\mathbf{f}}^{(t+1)}+{\mathbf{a}}^{(t+1)}\odot d{\mathbf{i}}^{(t+1)}+{\mathbf{i}}^{(t+1)}\odot d{\mathbf{a}}^{(t+1)}\\ & ={\mathbf{f}}^{(t+1)}\odot d{\mathbf{C}}^{(t)}+{\mathbf{C}}^{(t)}\odot {\mathbf{f}}^{(t+1)}\odot \left(1-{\mathbf{f}}^{(t+1)}\right)\odot d{\mathbf{W}}_f{\mathbf{h}}^{(t)}\\ & +{\mathbf{a}}^{(t+1)}\odot {\mathbf{i}}^{(t+1)}\odot \left(1-{\mathbf{i}}^{(t+1)}\right)\odot d{\mathbf{W}}_i{\mathbf{h}}^{(t)}\\ & +{\mathbf{i}}^{(t+1)}\odot \left(1-tan{h}^2({\mathbf{a}}^{(t+1)})\right)\odot d{\mathbf{W}}_a{\mathbf{h}}^{(t)}\\ & ={\mathbf{f}}^{(t+1)}\odot d{\mathbf{C}}^{(t)}+diag\left[{\mathbf{C}}^{(t)}\odot {\mathbf{f}}^{(t+1)}\odot \left(1-{\mathbf{f}}^{(t+1)}\right)\right]{\mathbf{W}}_{f}d{\mathbf{h}}^{(t)}\\ & +diag\left[{\mathbf{a}}^{(t+1)}\odot {\mathbf{i}}^{(t+1)}\odot \left(1-{\mathbf{i}}^{(t+1)}\right)\right]{\mathbf{W}}_{i}d{\mathbf{h}}^{(t)}\\ & +diag\left[{\mathbf{i}}^{(t+1)}\odot \left(1-tan{h}^2({\mathbf{a}}^{(t+1)})\right)\right]{\mathbf{W}}_{a}d{\mathbf{h}}^{(t)}\end{array}$$
将$d{\mathbf{C}}^{(t+1)}$代入$d{\mathbf{h}}^{(t+1)}$得到一个很庞大的式子，你们可以自己计算，这边直接给出答案：
$$\begin{array}{rl}\frac{\partial {\mathbf{h}}^{(t+1)}}{\partial {\mathbf{h}}^{(t)}}& ={\left({\mathbf{W}}_o\right)}^{T}diag\left[tanh({\mathbf{C}}^{(t+1)})\odot {\mathbf{o}}^{(t+1)}\odot (1-{\mathbf{o}}^{(t+1)})\right]+{\left({\mathbf{W}}_f\right)}^{T}diag\left[{\mathbf{C}}^{(t)}\odot {\mathbf{f}}^{(t+1)}\odot \left(1-{\mathbf{f}}^{(t+1)}\right)\odot \Delta \mathbf{C}\right]\\ & +{\left({\mathbf{W}}_i\right)}^{T}diag\left[{\mathbf{a}}^{(t+1)}\odot {\mathbf{i}}^{(t+1)}\odot \left(1-{\mathbf{i}}^{(t+1)}\right)\odot \Delta \mathbf{C}\right]+{\left({\mathbf{W}}_a\right)}^{T}diag\left[{\mathbf{i}}^{(t+1)}\odot \left(1-tan{h}^2({\mathbf{a}}^{(t+1)})\right)\odot \Delta \mathbf{C}\right]\\ 其中，\Delta \mathbf{C}& ={\mathbf{o}}^{(t+1)}\odot \left(1-tan{h}^2({\mathbf{C}}^{(t+1)})\right)\end{array}$$
有了${\mathbf{\delta }}_h^{(t)}$，${\mathbf{\delta }}_C^{(t)}$就很容易得出来了：

有了${\mathbf{\delta }}_h^{(t)},{\mathbf{\delta }}_C^{(t)}$，其他一些参数的梯度就很容易得出来了。