深度学习笔记之LSTM轻量级变体——门控循环单元[GRU]
引言
上一节介绍了从反向传播过程的角度认识 LSTM \text{LSTM} LSTM如何抑制梯度消失的问题。本节以 LSTM \text{LSTM} LSTM的问题为引,介绍它的一种轻量级变体——门控循环单元。
回顾: LSTM \text{LSTM} LSTM的前馈计算过程
关于 LSTM \text{LSTM} LSTM的结构展开图与前馈计算过程表示如下:
对应一个单元 ( Cell ) (\text{Cell}) (Cell)的前馈计算过程:
{ f ( t ) = σ [ W H ⇒ F ⋅ h ( t − 1 ) + W X ⇒ F ⋅ x ( t ) + b F ] i ( t ) = σ [ W H ⇒ I ⋅ h ( t − 1 ) + W X ⇒ I ⋅ x ( t ) + b I ] C ~ ( t ) = Tanh [ W H ⇒ C ~ ⋅ h ( t − 1 ) + W X ⇒ C ~ ⋅ x ( t ) + b C ~ ] C ( t ) = f ( t ) ∗ C ( t − 1 ) + i t ∗ C ~ ( t ) o ( t ) = σ [ W H ⇒ O ⋅ h ( t − 1 ) + W X ⇒ O ⋅ x ( t ) + b O ] h ( t ) = o ( t ) ∗ Tanh ( C ( t ) ) \begin{cases} \begin{aligned} & f^{(t)} = \sigma \left[\mathcal W_{\mathcal H \Rightarrow \mathcal F} \cdot h^{(t-1)} + \mathcal W_{\mathcal X \Rightarrow \mathcal F} \cdot x^{(t)} + b_{\mathcal F}\right] \\ & i^{(t)} = \sigma \left[\mathcal W_{\mathcal H \Rightarrow \mathcal I} \cdot h^{(t-1)} + \mathcal W_{\mathcal X \Rightarrow \mathcal I} \cdot x^{(t)} + b_{\mathcal I}\right] \\ & \widetilde{\mathcal C}^{(t)} = \text{Tanh} \left[\mathcal W_{\mathcal H \Rightarrow \widetilde{\mathcal C}} \cdot h^{(t-1)} + \mathcal W_{\mathcal X \Rightarrow \widetilde{\mathcal C}} \cdot x^{(t)} + b_{\widetilde{\mathcal C}}\right] \\ & \mathcal C^{(t)} = f^{(t)} * \mathcal C^{(t-1)} + i_t * \widetilde{\mathcal C}^{(t)} \\ & o^{(t)} = \sigma \left[\mathcal W_{\mathcal H \Rightarrow \mathcal O} \cdot h^{(t-1)} + \mathcal W_{\mathcal X \Rightarrow \mathcal O} \cdot x^{(t)} + b_{\mathcal O}\right] \\ & h^{(t)} = o^{(t)} * \text{Tanh}(\mathcal C^{(t)}) \end{aligned} \end{cases} ⎩
⎨
⎧f(t)=σ[WH⇒F⋅h(t−1)+WX⇒F⋅x(t)+bF]i(t)=σ[WH⇒I⋅h(t−1)+WX⇒I⋅x(t)+bI]C
(t)=Tanh[WH⇒C
⋅h(t−1)+WX⇒C
⋅x(t)+bC
]C(t)=f(t)∗C(t−1)+it∗C
(t)o(t)=σ[WH⇒O⋅h(t−1)+WX⇒O⋅x(t)+bO]h(t)=o(t)∗