Regularizing your neural network 神经网络正则化
Logistic regression regularization
先用简单的逻辑回归正则化作为例子,因为神经网络的参数 W W 是2维的。
无正则
L2 L 2 正则
J(w,b)=1m∑i=1mL(y^(i)−y(i))+λ2m||w||22 J ( w , b ) = 1 m ∑ i = 1 m L ( y ^ ( i ) − y ( i ) ) + λ 2 m | | w | | 2 2
||w||22=∑j=1nxw2j=wTw | | w | | 2 2 = ∑ j = 1 n x w j 2 = w T w-
L1
L
1
正则
J(w,b)=1m∑i=1mL(y^(i)−y(i))+λm||w||1 J ( w , b ) = 1 m ∑ i = 1 m L ( y ^ ( i ) − y ( i ) ) + λ m | | w | | 1 - ||w||1=∑j=1nx|w|j | | w | | 1 = ∑ j = 1 n x | w | j
Neural network regularization
- Frobenius正则(类似
L2
L
2
正则)
J(w[1],b[1],⋯,w[l],b[l])=1m∑i=1mL(y^(i),y(i))+12m∑l=1L||w[l]||2F J ( w [ 1 ] , b [ 1 ] , ⋯ , w [ l ] , b [ l ] ) = 1 m ∑ i = 1 m L ( y ^ ( i ) , y ( i ) ) + 1 2 m ∑ l = 1 L | | w [ l ] | | F 2
||w[l]||2F=∑i=1n[l]∑j=1n[l−1](w[l]ij)2 | | w [ l ] | | F 2 = ∑ i = 1 n [ l ] ∑ j = 1 n [ l − 1 ] ( w i j [ l ] ) 2
相较于无正则化的反向传播,正则化的反向传播在更新 W W 时,会对其进行权重衰减(weight decay),并下降。
w[l]:=w[l]−αdw[l]=w[l]−αλmw[l]−α(from backpropagation)=(1−αλm)w[l]−α(from backpropagation) w [ l ] : = w [ l ] − α d w [ l ] = w [ l ] − α λ m w [ l ] − α ( f r o m b a c k p r o p a g a t i o n ) = ( 1 − α λ m ) w [ l ] − α ( f r o m b a c k p r o p a g a t i o n )
- Frobenius正则(类似
L2
L
2
正则)