LSTM如何解决梯度消失与梯度爆炸

  这是一张经典的LSTM示意图，LSTM依靠 $f_t$、$i_t$、$o_t$来控制输入输出，$$f_t=\sigma \left(W_f\cdot \left[h_{t-1},x_t\right]+b_f\right)$$$$i_t=\sigma \left(W_i\cdot \left[h_{t-1},x_t\right]+b_i\right)$$$$o_t=\sigma \left(W_o\left[h_{t-1},x_t\right]+b_o\right)$$
  我们将其简化为：$$f_t=\sigma \left(W_f{X}_t+b_f\right)$$$$i_t=\sigma \left(W_i{X}_t+b_i\right)$$$$o_i=\sigma \left(W_o{X}_t+b_o\right)$$
  当前的状态 $S_t=f_t{S}_{t-1}+i_t{X}_t$ 类似与传统RNN  $S_t=W_s{S}_{t-1}+W_x{X}_t+b_1$ 。将LSTM的状态表达式展开后得：$$S_t=\sigma \left(W_f{X}_t+b_f\right)S_{t-1}+\sigma \left(W_i{X}_t+b_i\right)X_t$$  如果加上激活函数$$S_t=\tanh \left[\sigma \left(W_f{X}_t+b_f\right)S_{t-1}+\sigma \left(W_i{X}_t+b_i\right)X_t\right]$$  RNN梯度消失和爆炸的原因这篇文章中传统RNN求偏导的过程包含：$$\prod _{j=k+1}^t\frac{\partial S_j}{\partial S_{j-1}}=\prod _{j=k+1}^t{\tanh }^{\prime }{W}_s$$  对于LSTM同样也包含这样的一项，但是在LSTM中：$$\prod _{j=k+1}^t\frac{\partial S_j}{\partial S_{j-1}}=\prod _{j=k+1}^t{\tanh }^{\prime }\sigma \left(W_f{X}_t+b_f\right)$$ 假设  $Z={\tanh }^{\prime }(x)\sigma (y)$，则$Z$的函数图像如下图所示：

  可以看到该函数值基本上不是0就是1。
  传统RNN的求偏导过程：$$\frac{\partial L_3}{\partial W_s}=\frac{\partial L_3}{\partial O_3}\frac{\partial O_3}{\partial S_3}\frac{\partial S_3}{\partial W_s}+\frac{\partial L_3}{\partial O_3}\frac{\partial O_3}{\partial S_3}\frac{\partial S_3}{\partial S_2}\frac{\partial S_2}{\partial W_s}+\frac{\partial L_3}{\partial O_3}\frac{\partial O_3}{\partial S_3}\frac{\partial S_3}{\partial S_2}\frac{\partial S_2}{\partial S_1}\frac{\partial S_1}{\partial W_s}$$
  在LSTM中为：$$\frac{\partial L_3}{\partial W_s}=\frac{\partial L_3}{\partial O_3}\frac{\partial O_3}{\partial S_3}\frac{\partial S_3}{\partial W_s}+\frac{\partial L_3}{\partial O_3}\frac{\partial O_3}{\partial S_3}\frac{\partial S_2}{\partial W_s}+\frac{\partial L_3}{\partial O_3}\frac{\partial O_3}{\partial S_3}\frac{\partial S_1}{\partial W_s}$$
  因为$$\prod _{j=k+1}^t\frac{\partial S_j}{\partial S_{j-1}}=\prod _{j=k+1}^t{\tanh }^{\prime }\sigma \left(W_f{X}_t+b_f\right)\approx 0|1$$
  这样就解决了传统RNN中梯度消失的问题。