RNN 训练算法 —— BPDC (Backpropagation-Decorrelation)

问题描述

考虑模型循环网络模型：
$$x(k+1)=(1-\Delta t)x(k)+\Delta tWf[x(k)]\text{\hspace{1em}(1)}$$
其中 $x(k)\in R^N$表示网络节点在激活前的状态，$W\in R^{N\times N}$表示网络结点之间相互连接的权重，网络的输出节点为 { x i ( k ) ∣ i ∈ O } \{x_i(k)| i\in O\} {xi​(k)∣i∈O}，$O$为所有输出（或称“观测”）单元的下标集合

训练的目标是为了减少观测状态和预期值之间误差，即最小化损失函数：
$$E=\frac{1}{2}\sum _{k=1}^K\sum _{i\in O}[x_i(k)-d_i(k){]}^2\text{\hspace{1em}(2)}$$
其中 $d_i(k)$ 表示 $k$ 时刻第 $i$ 个节点的预期值

符号约定

$$W\equiv {\left[\begin{array}{c}\text{—–}{w}_1^T\text{—–}\\ ⋮\\ \text{—–}{w}_N^T\text{—–}\end{array}\right]}_{N\times N}$$
将矩阵 $W$ 拉成列向量，记为 $w$
$$w=[w_1^T,\cdots ,w_N^T{]}^T\in R^{N^2}$$
把所有时间的状态拼成列向量，记为 $x$
$$x=[x^T(1),\cdots ,x^T(K){]}^T\in R^{NK}$$
将RNN 的训练视为约束优化问题，(1)式转化成约束条件:
$$g(k+1)\equiv -x(k+1)+(1-\Delta t)x(k)+\Delta tWf[x(k)],k=1,\dots ,K\text{\hspace{1em}(3)}$$
记
$$g=[g^T(1),\dots ,g^T(K){]}^T\in R^{NK}$$

Atiya-Parlos 算法回顾

以上是经典的梯度下降法的思维，但是 Atiya-Parlos 提出了另一种优化思路：不是朝着参数的梯度方向更新，但仍使代价函数下降

该算法的思想是互换网络状态 $x(k)$ 和权重矩阵 $W$ 的作用：将状态视为控制变量，并根据 $x(k)$ 的变化确定权重的变化。 换句话说，我们计算 $E$ 相对于状态 $x(k)$ 的梯度，并假设状态在该梯度的负方向 $\Delta x_i(k)=-\eta \frac{\partial E}{\partial x_i(k)}$ 上有微小变化。

接下来，我们$确定权重W的变化\Delta w，以使由权重变化导致的状态变化尽可能地接近目标变化\Delta x$

该算法的细节如下：
$$\begin{array}{rl}\Delta x& =-\eta {\left(\frac{\partial E}{\partial x_i}\right)}^T\\ & =-\eta e^T\\ & =-\eta [e(1),\dots ,e(K){]}^T\\ & \\ e_i(k)& =\{\begin{array}{ll}{x}_i(k)-d_i(k),& \text{if }i\in O,\\ 0,& \text{otherwise. }\end{array}k\in 1,\dots ,K.\end{array}$$

由约束条件得：
$$\frac{\partial g}{\partial x}\Delta x=-\frac{\partial g}{\partial w}\Delta w$$
故已知 $\Delta x$ 时，可得：
$$\Delta w=-{\left[{\left(\frac{\partial g}{\partial w}\right)}^T\left(\frac{\partial g}{\partial w}\right)\right]}^{-1}{\left(\frac{\partial g}{\partial w}\right)}^T\left(\frac{\partial g}{\partial x}\right)\Delta x$$
需要注意逆矩阵不一定存在，故
$$\Delta w=-{\left[{\left(\frac{\partial g}{\partial w}\right)}^T\left(\frac{\partial g}{\partial w}\right)+ϵI\right]}^{-1}{\left(\frac{\partial g}{\partial w}\right)}^T\left(\frac{\partial g}{\partial x}\right)\Delta x$$
这就是权重 $W$ 的更新规则

计算细节

计算 $\frac{\partial g}{\partial w}$
$$\frac{\partial g}{\partial w}=\left[\begin{array}{c}\frac{\partial g(1)}{\partial w}\\ ⋮\\ \frac{\partial g(K)}{\partial w}\end{array}\right]=\Delta t\left[\begin{array}{c}\frac{\partial Wf[x(0)]}{\partial w}\\ ⋮\\ \frac{\partial Wf[x(K-1)]}{\partial w}\end{array}\right]$$
其中
$$\begin{array}{rl}\frac{\partial Wf[x(k)]}{\partial w}& =\left[\begin{array}{c}\frac{\partial w_1^{T}f[x(k)]}{\partial w}\\ ⋮\\ \frac{\partial w_N^{T}f[x(k)]}{\partial w}\end{array}\right]记f_k=[f(x_1),\dots ,f(x_N(k)){]}^T\\ & \\ & ={\left[\begin{array}{cccc}{f}_k^T& & & \\ & f_k^T& & \\ & & \ddots & \\ & & & f_k^T\end{array}\right]}_{N\times N^2}\\ & \\ & \triangleq F(k)\end{array}$$
故
$$\frac{\partial g}{\partial w}=\Delta t{\left[\begin{array}{c}F(0)\\ ⋮\\ F(K-1)\end{array}\right]}_{NK\times N^2}$$
1 Δ t 2 ( ∂ g ∂ w ) T ( ∂ g ∂ w ) = [ F T ( 0 ) ⋯ F T ( K − 1 ) ] [ F ( 0 ) ⋮ F ( K − 1 ) ] = ∑ k = 0 K − 1 F T ( k ) F ( k ) = [ ∑ k = 0 K − 1 f k f k T ∑ k = 0 K − 1 f k f k T ⋱ ∑ k = 0 K − 1 f k f k T ] N 2 × N 2 ≜ d i a g { C K − 1 } \begin{aligned} &\frac{1}{\Delta t^2}\left(\frac{\partial g}{\partial w}\right)^T \left(\frac{\partial g}{\partial w}\right) \\ &= \begin{bmatrix} F^T(0) & \cdots & F^T(K-1) \end{bmatrix} \begin{bmatrix} F(0)\\ \vdots \\ F(K-1) \end{bmatrix} \\\\ &= \sum_{k=0}^{K-1} F^T(k)F(k) \\\\ &=\begin{bmatrix} \sum_{k=0}^{K-1} f_k f_k^T &&& \\ & \sum_{k=0}^{K-1} f_k f_k^T && \\ && \ddots & \\ &&& \sum_{k=0}^{K-1} f_k f_k^T \end{bmatrix}_{N^2 \times N^2} \\\\ &\triangleq diag\{C_{K-1}\} \end{aligned} ​Δt21​(∂w∂g​)T(∂w∂g​)=[FT(0)​⋯​FT(K−1)​]⎣⎢⎡​F(0)⋮F(K−1)​⎦⎥⎤​=k=0∑K−1​FT(k)F(k)=⎣⎢⎢⎡​∑k=0K−1​fk​fkT​​∑k=0K−1​fk​fkT​​⋱​∑k=0K−1​fk​fkT​​⎦⎥⎥⎤​N2×N2​≜diag{CK−1​}​

令
$$\gamma ={\left[\begin{array}{c}\gamma (1)\\ \gamma (2)\\ ⋮\\ \gamma (K)\end{array}\right]}_{NK}=\frac{\partial g}{\partial x}\Delta x$$
$\gamma$ 表示由 $\Delta x$ 提供的误差信息，它的计算放在本文最后，先假设它已经求出来了

则
$$\begin{array}{rl}& {\left(\frac{\partial g}{\partial w}\right)}^T\left(\frac{\partial g}{\partial x}\right)\Delta x\\ & =\Delta t{\left[\begin{array}{ccc}{F}^T(0)& \cdots & F^T(K-1)\end{array}\right]}_{N^2\times NK}{\left[\begin{array}{c}\gamma (1)\\ \gamma (2)\\ ⋮\\ \gamma (K)\end{array}\right]}_{NK}\\ & =\Delta t\sum _{k=1}^K{F}^T(k-1)\gamma (k)\\ & =\Delta t\sum _{k=1}^K{\left[\begin{array}{cccc}{f}_{k-1}& & & \\ & f_{k-1}& & \\ & & \ddots & \\ & & & f_{k-1}\end{array}\right]}_{N^2\times N}{\left[\begin{array}{c}{\gamma }_1(k)\\ {\gamma }_2(k)\\ ⋮\\ {\gamma }_N(k)\end{array}\right]}_N\\ & \\ & =\Delta t{\left[\begin{array}{c}{\sum }_{k=1}^K{f}_{k-1}{\gamma }_1(k)\\ {\sum }_{k=1}^K{f}_{k-1}{\gamma }_2(k)\\ ⋮\\ {\sum }_{k=1}^K{f}_{k-1}{\gamma }_N(k)\end{array}\right]}_{N^2}\\ & \end{array}$$

所以
$$\begin{array}{rl}\Delta w& =-{\left[{\left(\frac{\partial g}{\partial w}\right)}^T\left(\frac{\partial g}{\partial w}\right)+ϵI\right]}^{-1}{\left(\frac{\partial g}{\partial w}\right)}^T\left(\frac{\partial g}{\partial x}\right)\Delta x\\ & =-\frac{1}{\Delta t}{\left[\begin{array}{c}{C}_{K-1}^{-1}{\sum }_{k=1}^K{f}_{k-1}{\gamma }_1(k)\\ C_{K-1}^{-1}{\sum }_{k=1}^K{f}_{k-1}{\gamma }_2(k)\\ ⋮\\ C_{K-1}^{-1}{\sum }_{k=1}^K{f}_{k-1}{\gamma }_N(k)\end{array}\right]}_{N^2}\\ & \\ \Delta W& =-\frac{1}{\Delta t}{\left[\begin{array}{c}{\sum }_{k=1}^K{f}_{k-1}^T{C}_{K-1}^{-1}{\gamma }_1(k)\\ {\sum }_{k=1}^K{f}_{k-1}^T{C}_{K-1}^{-1}{\gamma }_2(k)\\ ⋮\\ {\sum }_{k=1}^K{f}_{k-1}^T{C}_{K-1}^{-1}{\gamma }_N(k)\end{array}\right]}_{N\times N}\\ & =-\frac{1}{\Delta t}\sum _{k=1}^K{\left[\begin{array}{c}{f}_{k-1}^T{\gamma }_1(k)\\ f_{k-1}^T{\gamma }_2(k)\\ ⋮\\ f_{k-1}^T{\gamma }_N(k)\end{array}\right]}_{N\times N}{C}_{K-1}^{-1}\end{array}$$其中
$$C_{K-1}=ϵI+\sum _{r=0}^{K-1}{f}_r{f}_r^T$$
$注意：上述\Delta W是基于1,2,\dots ,K整个时间段的更新，不妨称之为\Delta W_{batch}$

下面将更新公式拆解在线更新（online updating）的形式：
$$\Delta W^{batch}(K)=\Delta W(1)+\cdots +\Delta W(K)$$

等式右端对应每一时刻的更新量

在第 $K$ 时刻的第 $i$ 个神经元的输入权重的更新量：
$$\begin{array}{rl}\Delta w_i^T(K)& =-\frac{1}{\Delta t}\sum _{k=1}^K{f}_{k-1}^T{C}_{K-1}^{-1}{\gamma }_i(k)+\frac{1}{\Delta t}\sum _{k=1}^{K-1}{f}_{k-1}^T{C}_{K-2}^{-1}{\gamma }_i(k)\\ & \\ & =-\frac{1}{\Delta t}{f}_{K-1}^T{C}_{K-1}^{-1}{\gamma }_i(K)-\frac{1}{\Delta t}\sum _{k=1}^{K-1}{f}_{k-1}^T(C_{K-1}^{-1}-C_{K-2}^{-1}){\gamma }_i(k)\\ & \\ & =-\frac{1}{\Delta t}{f}_{K-1}^T{C}_{K-1}^{-1}{\gamma }_i(K)-\frac{1}{\Delta t}\sum _{k=1}^{K-1}{f}_{k-1}^T{C}_{K-2}^{-1}{\gamma }_i(k)(C_{K-2}{C}_{K-1}^{-1}-I)\\ & \\ & =-\frac{1}{\Delta t}{f}_{K-1}^T{C}_{K-1}^{-1}{\gamma }_i(K)-\Delta w_i^{batch}(K-1)(C_{K-2}{C}_{K-1}^{-1}-I)\\ & \\ & =-\frac{1}{\Delta t}{f}_{K-1}^T{C}_{K-1}^{-1}{\gamma }_i(K)-\sum _{k=1}^{K-1}\Delta w_i^T(k)(C_{K-2}{C}_{K-1}^{-1}-I)\end{array}$$
可以看出，APRL 的更新规则由当前时刻的误差和 w 的累计更新（动量）组成

随着 $K\to \infty$，易知${\sum }_{k=1}^{K-1}\Delta w_i^T(k)\to const,C_{K-2}{C}_{K-1}^{-1}\to I$，所以第二项趋于零

BPDC 更新规则

BPDC 对 APRL 的在线算法做了简单粗暴的近似

该近似不试图累积完整的相关矩阵 $C_k$，也舍弃了先前误差的累积，而且只计算瞬时相关 $C(k)$：
$$\begin{array}{rl}\Delta w_i^T(k+1)& =-\frac{1}{\Delta t}{f}_k^{T}C(k{)}^{-1}{\gamma }_i(k+1)\\ & \\ C(k)& =ϵI+f_k{f}_k^T\end{array}$$
利用矩阵求逆引理：
$$\begin{array}{rl}C(k{)}^{-1}& =(ϵI+f_k{f}_k^T{)}^{-1}\\ & =\frac{1}{ϵ}I-\frac{1}{ϵ}\frac{f{f}^T}{ϵ+f^{T}f}\end{array}$$
所以
$$\begin{array}{rl}\Delta w_i^T(k+1)& =-\frac{1}{\Delta t}{f}_k^T\left(\frac{1}{ϵ}I-\frac{1}{ϵ}\frac{f{f}^T}{ϵ+f^{T}f}\right){\gamma }_i(k+1)\\ & \\ & =-\frac{1}{\Delta t}\frac{f^T}{ϵ+f^{T}f}{\gamma }_i(k+1)\end{array}$$

计算 $\gamma$

$$\gamma ={\left[\begin{array}{c}\gamma (1)\\ \gamma (2)\\ ⋮\\ \gamma (K)\end{array}\right]}_{NK}=\frac{\partial g}{\partial x}\Delta x=-\eta \frac{\partial g}{\partial x}[e(1),\dots ,e(K){]}^T$$
关键在与计算 $\frac{\partial g}{\partial x}$
$$\begin{array}{rl}\frac{\partial g}{\partial x}& =\left[\begin{array}{ccc}\frac{\partial g_1}{\partial x(1)}& \dots & \frac{\partial g_1}{\partial x(K)}\\ ⋮& \ddots & ⋮\\ \frac{\partial g_K}{\partial x(1)}& \dots & \frac{\partial g_K}{\partial x(K)}\end{array}\right]\\ & \\ & =\left[\begin{array}{ccccc}\frac{\partial g_1}{\partial x(1)}& 0& 0& \dots & 0\\ \frac{\partial g_2}{\partial x(1)}& \frac{\partial g_2}{\partial x(2)}& 0& \dots & 0\\ 0& \frac{\partial g_3}{\partial x(2)}& \frac{\partial g_3}{\partial x(3)}& \dots & 0\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 0& 0& 0& \frac{\partial g_K}{\partial x(K-1)}& \frac{\partial g_K}{\partial x(K)}\end{array}\right]\\ & \\ & =\left[\begin{array}{ccccc}-I& 0& 0& \dots & 0\\ (1-\Delta t)I+\Delta tWD(1)& -I& 0& \dots & 0\\ 0& (1-\Delta t)I+\Delta tWD(2)& -I& \dots & 0\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 0& 0& 0& (1-\Delta t)I+\Delta tWD(K-1)& -I\end{array}\right]\end{array}$$
其中
$$D(k)={\left[\begin{array}{ccc}{f}^{\prime }(x_1(k))& \cdots & 0\\ ⋮& \ddots & ⋮\\ 0& \dots & f^{\prime }(x_N(k))\end{array}\right]}_{N\times N}$$
所以
$$\begin{array}{rl}\gamma & =-\eta \frac{\partial g}{\partial x}[e(1),\dots ,e(K){]}^T\\ & =-\eta \left[\begin{array}{ccccc}-I& 0& 0& \dots & 0\\ (1-\Delta t)I+\Delta tWD(1)& -I& 0& \dots & 0\\ 0& (1-\Delta t)I+\Delta tWD(2)& -I& \dots & 0\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 0& 0& 0& (1-\Delta t)I+\Delta tWD(K-1)& -I\end{array}\right]\left[\begin{array}{c}e(1)\\ e(2)\\ e(3)\\ ⋮\\ e(K)\end{array}\right]\\ & \\ & =-\eta {\left[\begin{array}{c}-e(1)\\ [(1-\Delta t)I+\Delta tWD(1)]e(1)-e(2)\\ [(1-\Delta t)I+\Delta tWD(2)]e(2)-e(3)\\ ⋮\\ [(1-\Delta t)I+\Delta tWD(K-1)]e(K-1)-e(K)\end{array}\right]}_{NK\times 1}\end{array}$$
代入到 BPDC 更新规则：
Δ w i T ( k + 1 ) = − 1 Δ t f T ϵ + f T f γ i ( k + 1 ) = η Δ t f T ϵ + f T f { ( 1 − Δ t ) e i ( k ) + Δ t ∑ s ∈ O w i s f ′ ( x s ( k ) ) e s ( k ) − e i ( k + 1 ) } \begin{aligned} \Delta w^T_{i}(k+1) &= - \frac{1}{\Delta t} \frac{f^T}{\epsilon + f^Tf} \color{red}{ \gamma_i(k+1)} \\ &= \frac{\color{red}{\eta}}{\Delta t} \frac{f^T}{\epsilon + f^Tf} \color{red}\{ (1-\Delta t )e_i(k) + \Delta t \sum_{s\in O}w_{is} f'(x_s(k))e_s(k) - e_i(k+1) \} \end{aligned} ΔwiT​(k+1)​=−Δt1​ϵ+fTffT​γi​(k+1)=Δtη​ϵ+fTffT​{(1−Δt)ei​(k)+Δts∈O∑​wis​f′(xs​(k))es​(k)−ei​(k+1)}​

参考文献

• J.J. Steil, Backpropagation-decorrelation: online recurrent learning with O(N) complexity, in: Proceedings of the International Joint Conference on Neural Networks (IJCNN), vol. 1, 2004, pp. 843–848.
• J.J. Steil, Online stability of backpropagation-decorrelation recurrent learning, Neurocomputing 69 (2006) 642–650.
• J.J. Steil, Online reservoir adaptation by intrinsic plasticity for backpropagation-decorrelation and echo state learning, Neural Networks 20 (3) (2007) 353–364.

作者简介

服了这个德国佬，看了他三篇论文里的 $\gamma$ 都推错了，最后还是在下面这个大姐的毕业论文里找到了正确的公式