机器学习中的各种范数与正则化
对于统计机器学习算法一般为缓解过拟合现象的发生需要在进行正则化操作,通过正则化以偏差的增加换取方差的减小,因此优秀的正则化操作即是在保证方差较小的情况下,偏差尽可能的小。有关偏差与方差的介绍可以参考我的这篇笔记
PoderLee:Bias - Variance Dilemma(偏差-方差窘境)zhuanlan.zhihu.com![f991242efd4afdaf248a581f365ac90f.png](https://i-blog.csdnimg.cn/blog_migrate/8c255f9cd4c7764616a8fff6d44dc376.jpeg)
最常见的方法即为在损失函数中引入矩阵范数,以对模型的复杂程度做出惩罚,即模型的权值参数(一般不考虑惩罚偏置,这主要是因为惩罚偏置首先不会有明显效果,其次惩罚偏置可能会造成模型欠拟合),其目标函数一般如下式所示:
![equation?tex=min%5Cquad+%5Ctilde+J%28%5Ctheta%3BX%2Cy%29%3D%5Cunderbrace%7BJ%28%5Ctheta%3BX%2Cy%29%7D_%7B%281%29%7D%2B%5Cunderbrace%7B%5Calpha+%5COmega%28%5Ctheta%29%7D_%7B%282%29%7D%5Ctag%7B1%7D](https://i-blog.csdnimg.cn/blog_migrate/6ba1bd1ebce799a7bc18190e861691bd.png)
上式中,第一项即为经验风险,第二项即为正则化项。其中
![equation?tex=%5Calpha%5Cgeq0](https://i-blog.csdnimg.cn/blog_migrate/5b91dc0a0c85bdf6bbd775c9afc3cd7d.png)
![equation?tex=%5Calpha%3D0](https://i-blog.csdnimg.cn/blog_migrate/34ae93e1c9fa2117623e24c054066d46.png)
![equation?tex=%5Calpha](https://i-blog.csdnimg.cn/blog_migrate/b5edc03934564b6fc6a72b7804435505.png)
![equation?tex=+L_2%3A%5Cqquad+%5COmega%28%5Ctheta%29%3D%5Cfrac12%7C%7Cw%7C%7C%5E2_2%5Ctag%7B2%7D](https://i-blog.csdnimg.cn/blog_migrate/6cfc3aa09f1b078b5a7418944c7d1894.png)
这里我们假设模型的偏置参数均为0,则参数
![equation?tex=%5Ctheta](https://i-blog.csdnimg.cn/blog_migrate/679f2e1de0bb439ac572e914fed0f16f.png)
![equation?tex=w](https://i-blog.csdnimg.cn/blog_migrate/1c7a330b82678d6b033d9748f5120af9.png)
![equation?tex=+%5Ctilde+J%28w%3BX%2Cy%29%3DJ%28w%3BX%2Cy%29%2B%5Cfrac%7B%5Calpha%7D%7B2%7Dw%5ETw%5Ctag%7B3%7D+](https://i-blog.csdnimg.cn/blog_migrate/21c1aac05b7e4f5b0819c78e4766b7b8.png)
对上式求其梯度有:
![equation?tex=+%5Cnabla_w%5Ctilde+J%28w%3BX%2Cy%29%3D%5Cnabla_wJ%28w%3BX%2Cy%29%2B%5Calpha+w%5Ctag%7B4%7D+](https://i-blog.csdnimg.cn/blog_migrate/c9a76c0b8a5278eb31c88529e7c648a6.png)
使用梯度下降法更新权重
![equation?tex=w](https://i-blog.csdnimg.cn/blog_migrate/1c7a330b82678d6b033d9748f5120af9.png)
![equation?tex=w](https://i-blog.csdnimg.cn/blog_migrate/1c7a330b82678d6b033d9748f5120af9.png)
![equation?tex=+w%5C%3B%5Cleftarrow%5C%3Bw-%5Cepsilon%28%5Cnabla_wJ%28w%3BX%2Cy%29%2B%5Calpha+w%29%3D%281-%5Cepsilon%5Calpha%29w-%5Cepsilon%5Cnabla_wJ%28w%3BX%2Cy%29%5Ctag%7B5%7D+](https://i-blog.csdnimg.cn/blog_migrate/dfd2b895427598884e4dd277c012315c.png)
从中可以看出每次权值
![equation?tex=w](https://i-blog.csdnimg.cn/blog_migrate/1c7a330b82678d6b033d9748f5120af9.png)
![equation?tex=%281-%5Cepsilon%5Calpha%29](https://i-blog.csdnimg.cn/blog_migrate/c543d9c0228b847e2f4d8b74632da5dd.png)
进一步地,这里令
![equation?tex=w%5E%2A%3Darg%5C%2Cmin_wJ%28w%29](https://i-blog.csdnimg.cn/blog_migrate/1c7a330b82678d6b033d9748f5120af9.png%5E%2A%3Darg%5C%2Cmin_wJ%28w%29)
![equation?tex=J%28w%3BX%2Cy%29](https://i-blog.csdnimg.cn/blog_migrate/d7429f0a51c0ca2a739f81162524769b.png)
![equation?tex=J%28w%29](https://i-blog.csdnimg.cn/blog_migrate/c5363d781571b468fb119f606a55cd23.png)
![equation?tex=w%5E%2A](https://i-blog.csdnimg.cn/blog_migrate/1c7a330b82678d6b033d9748f5120af9.png%5E%2A)
![equation?tex=J%28w%3BX%2Cy%29](https://i-blog.csdnimg.cn/blog_migrate/d7429f0a51c0ca2a739f81162524769b.png)
![equation?tex=w%5E%2A](https://i-blog.csdnimg.cn/blog_migrate/1c7a330b82678d6b033d9748f5120af9.png%5E%2A)
![equation?tex=J%28w%29](https://i-blog.csdnimg.cn/blog_migrate/c5363d781571b468fb119f606a55cd23.png)
![equation?tex=%5Chat+J%28w%29](https://i-blog.csdnimg.cn/blog_migrate/c5ff184a6db666182f87f6ab7833356d.png)
![equation?tex=%5Chat+J%28w%29%3DJ%28w%5E%2A%29%2B%5Cfrac12%28w-w%5E%2A%29%5ETH%28w-w%5E%2A%29%5Ctag%7B6%7D](https://i-blog.csdnimg.cn/blog_migrate/c5ff184a6db666182f87f6ab7833356d.png%3DJ%28w%5E%2A%29%2B%5Cfrac12%28w-w%5E%2A%29%5ETH%28w-w%5E%2A%29%5Ctag%7B6%7D)
上式中
![equation?tex=H](https://i-blog.csdnimg.cn/blog_migrate/3aaa0898c3912e86ae26b3d1ba3987fb.png)
![equation?tex=J](https://i-blog.csdnimg.cn/blog_migrate/487efbc22f7a0514b4b7934844e7bcc2.png)
![equation?tex=w%5E%2A](https://i-blog.csdnimg.cn/blog_migrate/1c7a330b82678d6b033d9748f5120af9.png%5E%2A)
![equation?tex=w%5E%2A](https://i-blog.csdnimg.cn/blog_migrate/1c7a330b82678d6b033d9748f5120af9.png%5E%2A)