- 梯度爆炸:梯度过大会使得损失函数很难收敛,甚至导致梯度为 NaN,异常退出;
- 梯度消失:较前的层次很难对较后的层次产生影响,梯度更新失效;
- 解决方案:对于 RNN 模型而言,采用 GRU(两个门控制) 或者更多门控制的 LSTM(forget gate、update gate、output gate)
1. 梯度消失与梯度爆炸
- gradient clipping
- 梯度爆炸:min(grad_max,grad)
- 梯度消失:max(grad_min, grad)
2. gradient clipping
def clip(gradients, maxValue=10):
'''
Clips the gradients' values between minimum and maximum.
Arguments:
gradients -- a dictionary containing the gradients "dWaa", "dWax", "dWya", "db", "dby"
Returns:
gradients -- a dictionary with the clipped gradients.
'''
dWaa, dWax, dWya, db, dby = gradients['dWaa'], gradients['dWax'],
gradients['dWya'], gradients['db'], gradients['dby']
for gradient in [dWax, dWaa, dWya, db, dby]:
np.clip(gradient, -maxValue, maxValue, out=gradient)
gradients = {"dWaa": dWaa, "dWax": dWax, "dWya": dWya, "db": db, "dby": dby}
return gradients