Chap 7深度学习中的正则化——L1正则化公式推导

对(7.22)到(7.23)进行一个推导（不保证严谨性）

已知：$H_{i,i}>0,\alpha >0,w_i在w_i^{\ast }某邻域U(w_i,\delta )内，\delta 充分小使得\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}(w_i)=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}(w_i^{\ast })$。
证明：
令
$$f_i(w_i)=\frac{1}{2}{H}_{i,i}(w_i-w_i^{\ast }{)}^2+\alpha |w_i|$$则
$$\hat{J}(w;X,y)=J(w^{\ast };X,y)+\sum _i{f}_i(w_i)(1)$$令
$$\frac{\partial \hat{J}}{\partial w_i}=\frac{\partial f_i}{\partial w_i}=0$$则
$$H_{i,i}(w_i-w_i^{\ast })+\alpha \mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}(w_i)=0(2)$$
(i) 当$w_i<0,\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}(w_i)=-1,代入(2),解得$
$$w_i=\frac{H_{i,i}{w}_i^{\ast }+\alpha }{H_{i,i}}=w_i^{\ast }+\frac{\alpha }{H_{i,i}}=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}(w_i^{\ast })(|w_i^{\ast }|-\frac{\alpha }{H_{i,i}})(3)$$
$$\because w_i<0\therefore w_i^{\ast }<-\frac{\alpha }{H_{i,i}}或|w_i^{\ast }|>\frac{\alpha }{H_{i,i}}$$
如$$-\frac{\alpha }{H_{i,i}}<w_i^{\ast }<0或|w_i^{\ast }|<\frac{\alpha }{H_{i,i}}$$
则$$\frac{\partial f_i}{\partial w_i}=H_{i,i}{w}_i-H_{i,i}{w}_i^{\ast }-\alpha <-H_{i,i}{w}_i^{\ast }-\alpha <0$$
(ii) 当$w_i>0,\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}(w_i)=1,代入(2),解得$
$$w_i=\frac{H_{i,i}{w}_i^{\ast }-\alpha }{H_{i,i}}=w_i^{\ast }-\frac{\alpha }{H_{i,i}}=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}(w_i^{\ast })(|w_i^{\ast }|-\frac{\alpha }{H_{i,i}})(4)$$
$$\because w_i>0\therefore w_i^{\ast }>\frac{\alpha }{H_{i,i}}或|w_i^{\ast }|>\frac{\alpha }{H_{i,i}}$$
如$$0<w_i^{\ast }<\frac{\alpha }{H_{i,i}}或|w_i^{\ast }|<\frac{\alpha }{H_{i,i}}$$
则$$\frac{\partial f_i}{\partial w_i}=H_{i,i}{w}_i-H_{i,i}{w}_i^{\ast }+\alpha >-H_{i,i}{w}_i^{\ast }+\alpha >0$$

由(i)(ii)(iii)，当$|w_i^{\ast }|\ge \frac{\alpha }{H_{i,i}}$，解析解$w_i=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}(w_i^{\ast })(|w_i^{\ast }|-\frac{\alpha }{H_{i,i}})$

当$|w_i^{\ast }|\le \frac{\alpha }{H_{i,i}}$,有
$$\frac{\partial f_i}{\partial w_i}=\{\begin{array}{ll}-H_{i,i}{w}_i^{\ast }-\alpha <0& w_i<0\\ -H_{i,i}{w}_i^{\ast }+\alpha >0& w_i>0\end{array}$$
故在负半轴偏导数单调递减，在正半轴偏导数单调递增，即在$w_i=0$处取得偏导数的极小值。

综上， w i = s i g n ( w i ∗ ) max ⁡ { ∣ w i ∗ ∣ − α H i , i , 0 } w_i=\mathrm{sign}(w_i^*)\max\{|w_i^*|-\dfrac{\alpha}{H_{i,i}},0\} wi​=sign(wi∗​)max{∣wi∗​∣−Hi,i​α​,0}
证毕。