Deep learning II - I Practical aspects of deep learning - Regularizing your neural network 神经网络范数正则化

Regularizing your neural network 神经网络正则化

---

Logistic regression regularization

先用简单的逻辑回归正则化作为例子，因为神经网络的参数$W$是2维的。

1. 无正则
   

   $$J(w,b) = \frac{1}{m} \sum_{i=1}^{m} {\cal L}(\hat{y}^{(i)} - y^{(i)})$$

   • $L_2$ 正则
     

     $$J(w,b) = \frac{1}{m} \sum_{i=1}^{m} {\cal L}(\hat{y}^{(i)} - y^{(i)}) + \frac{\lambda}{2m}||w||^2_2$$
     $$||w||^2_2 = \sum_{j=1}^{n_x}w_j^2 = w^Tw$$

   • $L_1$ 正则
     $$J(w,b) = \frac{1}{m} \sum_{i=1}^{m} {\cal L}(\hat{y}^{(i)} - y^{(i)}) + \frac{\lambda}{m}||w||_1$$

   • $$||w||_1 = \sum_{j=1}^{n_x}|w|_j$$

     Neural network regularization

     1. Frobenius正则(类似$L_2$正则)
        $$J(w^{[1]},b^{[1]}, \cdots , w^{[l]},b^{[l]}) = \frac{1}{m} \sum_{i =1}^m {\cal L}(\hat{y}^{(i)}, y^{(i)}) + \frac{1}{2m} \sum_{l=1}^L ||w^{[l]}||_F^2$$
        $$||w^{[l]}||_F^2 = \sum_{i = 1}^{n^{[l]}} \sum_{j = 1}^{n^{[l-1]}}(w^{[l]}_{ij})^2$$

     相较于无正则化的反向传播，正则化的反向传播在更新$W$时，会对其进行权重衰减（weight decay），并下降。
     

     $${\rm d}w^{[l]} = (from\ backpropagation) + \frac{\lambda}{m}w^{[l]}$$
     
     w[l]:=w[l]−αdw[l]=w[l]−αλmw[l]−α(from backpropagation)=(1−αλm)w[l]−α(from backpropagation) w [ l ] : = w [ l ] − α d w [ l ] = w [ l ] − α λ m w [ l ] − α ( f r o m   b a c k p r o p a g a t i o n ) = ( 1 − α λ m ) w [ l ] − α ( f r o m   b a c k p r o p a g a t i o n )
     $$\begin{split}w^{[l]}: &=w^{[l]} - \alpha {\rm d}w^{[l]} \\&= w^{[l]} - \alpha \frac{\lambda}{m}w^{[l]} - \alpha (from\ backpropagation) \\&= (1- \alpha \frac{\lambda}{m})w^{[l]} - \alpha (from\ backpropagation)\end{split}$$