RNN求导公式详细推导

本菜鸡觉得RNN求导公式太复杂了, 所以想了一个办法拆分求导的公式.

那就是用语法树.

原文参见RNN反向求导详解_格物致知-CSDN博客

$$\begin{array}{rl}{o}_t& =\phi (V{s}_t)=\phi (V\varphi (W{s}_{t-1}+U{x}_t))\\ L_t& =\text{loss}(o_t,y_t)\end{array}$$
令$o_t^{\ast }=V{s}_t$, $s_t^{\ast }=U{x}_t+W{s}_{t-1}$

则$o_t=\phi (o_t^{\ast })$, $s_t=\varphi (s_t^{\ast })$

现在把$L_t$画成一棵语法树, 然后开始一步一步求导

用$\ast$表示元素相乘, 用$\times$表示矩阵乘法
$$\begin{array}{r}\frac{\partial L_t}{\partial o_t^{\ast }}=\frac{\partial L_t}{\partial o_t}\ast \frac{\partial o_t}{\partial o_t^{\ast }}=\frac{\partial L_t}{\partial o_t}\ast {\phi }^{\prime }(o_t^{\ast })\end{array}\text{\hspace{1em}(1)}$$
式1的结果是一个与$o_t^{\ast }$的维度一致的向量.
$$\frac{\partial L_t}{\partial V_t}=\frac{\partial L_t}{\partial o_t^{\ast }}[?]\frac{\partial o_t^{\ast }}{\partial V}\text{\hspace{1em}(2)}$$
公式2整体上是标量对矩阵求导, 标量对矩阵求导就是标量对矩阵中的每个元素求导; 有一个中间值$o_t^{\ast }$是向量.

的前半部分在公式1中求过了, 后面是对矩阵×向量的求导

既然是对$V$求导那结果的形状必然跟$V$一样

还是写个例子算算怎么求导吧

$${\mathbf{o}}^{\ast }=\mathbf{V}\times \mathbf{s}=\left[\begin{array}{cccc}{V}_{11}& V_{12}& V_{13}& V_{14}\\ V_{21}& V_{22}& V_{23}& V_{24}\\ V_{31}& V_{32}& V_{33}& V_{34}\end{array}\right]\times \left[\begin{array}{c}{s}_1\\ s_2\\ s_3\\ s_4\end{array}\right]=\left[\begin{array}{c}{V}_{11}{s}_1+V_{12}{s}_2+V_{13}{s}_3+V_{14}{s}_4\\ V_{21}{s}_1+V_{22}{s}_2+V_{23}{s}_3+V_{24}{s}_4\\ V_{31}{s}_1+V_{32}{s}_2+V_{33}{s}_3+V_{34}{s}_4\end{array}\right]=\left[\begin{array}{c}{o}_1^{\ast }\\ o_2^{\ast }\\ o_3^{\ast }\end{array}\right]\text{\hspace{1em}(3)}$$
$$\begin{array}{rl}\frac{\partial L}{\partial V_{11}}=\frac{\partial L}{\partial o_1^{\ast }}\frac{\partial o_1^{\ast }}{\partial V_{11}}& =\frac{\partial L}{\partial o_1^{\ast }}{s}_1\\ \frac{\partial L}{\partial V_{12}}=\frac{\partial L}{\partial o_1^{\ast }}\frac{\partial o_1^{\ast }}{\partial V_{12}}& =\frac{\partial L}{\partial o_1^{\ast }}{s}_2\\ & ⋮\\ \frac{\partial L}{\partial V_{34}}=\frac{\partial L}{\partial o_3^{\ast }}\frac{\partial o_3^{\ast }}{\partial V_{34}}& =\frac{\partial L}{\partial o_3^{\ast }}{s}_4\end{array}$$
$$\begin{array}{r}\frac{\partial L}{\partial V}=\left[\begin{array}{c}\frac{\partial L}{\partial o_1^{\ast }}\\ \frac{\partial L}{\partial o_2^{\ast }}\\ \frac{\partial L}{\partial o_3^{\ast }}\end{array}\right]\times \left[\begin{array}{cccc}{s}_1& s_2& s_3& s_4\end{array}\right]\end{array}$$
所以式2应该写成
$$\frac{\partial L_t}{\partial V_t}=\frac{\partial L_t}{\partial o_t^{\ast }}\times \frac{\partial o_t^{\ast }}{\partial V}=\frac{\partial L_t}{\partial o_t^{\ast }}\times s_t^T\text{\hspace{1em}(4)}$$
然后求$L_t$对$s_t$的导数, 还要参考式3


图片来源: https://wenku.baidu.com/view/0c28ff2249d7c1c708a1284ac850ad02de8007c1.html

$$\begin{array}{rl}\frac{\partial L}{\partial s_1}& =\left[\begin{array}{c}\frac{\partial L}{\partial o_1^{\ast }}\frac{\partial o_1^{\ast }}{\partial s_1}+\frac{\partial L}{\partial o_2^{\ast }}\frac{\partial o_2^{\ast }}{\partial s_1}+\frac{\partial L}{\partial o_3^{\ast }}\frac{\partial o_3^{\ast }}{\partial s_1}\end{array}\right]\\ & ⋮\\ \frac{\partial L}{\partial s_4}& =\left[\begin{array}{c}\frac{\partial L}{\partial o_1^{\ast }}\frac{\partial o_1^{\ast }}{\partial s_4}+\frac{\partial L}{\partial o_2^{\ast }}\frac{\partial o_2^{\ast }}{\partial s_4}+\frac{\partial L}{\partial o_3^{\ast }}\frac{\partial o_3^{\ast }}{\partial s_4}\end{array}\right]\end{array}$$
$$\begin{array}{rl}\frac{\partial L}{\partial s}& =\left[\begin{array}{c}\frac{\partial L}{\partial o_1^{\ast }}\frac{\partial o_1^{\ast }}{\partial s_1}+\frac{\partial L}{\partial o_2^{\ast }}\frac{\partial o_2^{\ast }}{\partial s_1}+\frac{\partial L}{\partial o_3^{\ast }}\frac{\partial o_3^{\ast }}{\partial s_1}\\ \frac{\partial L}{\partial o_1^{\ast }}\frac{\partial o_1^{\ast }}{\partial s_2}+\frac{\partial L}{\partial o_2^{\ast }}\frac{\partial o_2^{\ast }}{\partial s_2}+\frac{\partial L}{\partial o_3^{\ast }}\frac{\partial o_3^{\ast }}{\partial s_2}\\ \frac{\partial L}{\partial o_1^{\ast }}\frac{\partial o_1^{\ast }}{\partial s_3}+\frac{\partial L}{\partial o_2^{\ast }}\frac{\partial o_2^{\ast }}{\partial s_3}+\frac{\partial L}{\partial o_3^{\ast }}\frac{\partial o_3^{\ast }}{\partial s_3}\\ \frac{\partial L}{\partial o_1^{\ast }}\frac{\partial o_1^{\ast }}{\partial s_4}+\frac{\partial L}{\partial o_2^{\ast }}\frac{\partial o_2^{\ast }}{\partial s_4}+\frac{\partial L}{\partial o_3^{\ast }}\frac{\partial o_3^{\ast }}{\partial s_4}\end{array}\right]\\ & =\left[\begin{array}{ccc}\frac{\partial o_1^{\ast }}{\partial s_1}& \frac{\partial o_2^{\ast }}{\partial s_1}& \frac{\partial o_3^{\ast }}{\partial s_1}\\ \frac{\partial o_1^{\ast }}{\partial s_2}& \frac{\partial o_2^{\ast }}{\partial s_2}& \frac{\partial o_3^{\ast }}{\partial s_2}\\ \frac{\partial o_1^{\ast }}{\partial s_3}& \frac{\partial o_2^{\ast }}{\partial s_3}& \frac{\partial o_3^{\ast }}{\partial s_3}\\ \frac{\partial o_1^{\ast }}{\partial s_4}& \frac{\partial o_2^{\ast }}{\partial s_4}& \frac{\partial o_3^{\ast }}{\partial s_4}\end{array}\right]\times \left[\begin{array}{c}\frac{\partial L}{\partial o_1^{\ast }}\\ \frac{\partial L}{\partial o_2^{\ast }}\\ \frac{\partial L}{\partial o_3^{\ast }}\end{array}\right]\\ & =?\times \frac{\partial L}{\partial o_t}\end{array}\text{\hspace{1em}(5)}$$
要解决式5的后一步, 需要先向量求导的问题

参考链接: https://zhuanlan.zhihu.com/p/36448789

文中有一句话:

不过为了方便我们在实践中应用，通常情况下即使$y$向量是列向量也按照行向量来进行求导。

根据这句话可以得出, 一般情况下是行向量对列向量求导.

行向量$X$对列向量$Y$求导会形成一个矩阵, 矩阵的宽度是$X$的长度, 矩阵的高度是$Y$的长度

所以式5中的问号矩阵应该是一个行向量$o_t^{\ast }$对列向量$s$求导
$$\frac{\partial L}{\partial s_t}=\frac{\partial o_t^{\ast }}{\partial s_t}\times \frac{\partial L}{\partial o_t^{\ast }}\text{\hspace{1em}(6)}$$
式6中的$\frac{\partial o_t^{\ast }}{\partial s_t}$还可以继续求出结果
$$\begin{array}{rl}\frac{\partial o_t^{\ast }}{\partial s_t}& =\left[\begin{array}{ccc}\frac{\partial o_1^{\ast }}{\partial s_1}& \frac{\partial o_2^{\ast }}{\partial s_1}& \frac{\partial o_3^{\ast }}{\partial s_1}\\ \frac{\partial o_1^{\ast }}{\partial s_2}& \frac{\partial o_2^{\ast }}{\partial s_2}& \frac{\partial o_3^{\ast }}{\partial s_2}\\ \frac{\partial o_1^{\ast }}{\partial s_3}& \frac{\partial o_2^{\ast }}{\partial s_3}& \frac{\partial o_3^{\ast }}{\partial s_3}\\ \frac{\partial o_1^{\ast }}{\partial s_4}& \frac{\partial o_2^{\ast }}{\partial s_4}& \frac{\partial o_3^{\ast }}{\partial s_4}\end{array}\right]\\ & =\left[\begin{array}{ccc}{V}_{11}& V_{21}& V_{31}\\ V_{12}& V_{22}& V_{32}\\ V_{13}& V_{23}& V_{33}\\ V_{14}& V_{24}& V_{34}\end{array}\right]\\ & =V^T\end{array}$$
上面的结果带入式6中得到
$$\frac{\partial L}{\partial s_t}=\frac{\partial o_t^{\ast }}{\partial s_t}\times \frac{\partial L}{\partial o_t^{\ast }}=V^T\times \frac{\partial L}{\partial o_t^{\ast }}\text{\hspace{1em}(7)}$$
到此为止, 所以涉及到的技术都已经写完了, 把求导结果都填到语法树上后

分析后面发现, 后面的结构都是对前面的规律的简单重复.

所以后面随便填两个吧!