l2正则化python_回归分析_L2正则化（岭回归）【python实现】

样本

\[x_i=(x_{i1};x_{i2}; ...; x_{im}) \, {函数值 y_i}

\]

每个样本有m个变量

回归面

\[f(x_i) = x_i^T \omega +b

\]

\(\omega = (\omega_1; \omega_2; ...; \omega_m)\)

\[\hat x_i=(x_{i1};x_{i2}; ...; x_{im}; 1)

\]

\[\hat \omega = (\omega_1; \omega_2; ...; \omega_m; b)

\]

则

\[f(x_i) = \hat x_i^T \hat \omega

\]

假设有n个样本令

\[X = [\hat x_1^T; \hat x_2^T; ..., \hat x_n^T]

\]

则均方误差为

\[\frac 1n (X \hat \omega - Y)^T (X \hat \omega - Y)

\]

其中\(Y = (y_1; y_2; ...; y_n)\)

损失函数

\[J(\hat \omega^*) = (X \hat \omega^* - Y)^T (X\hat \omega^* - Y)

\]

令

\[\left. \frac{{\rm d} J(\hat \omega^*)}{{\rm d} \hat \omega^*} = 0\right.

\]

\[\hat \omega^* = (X^TX)^{-1}X^TY

\]

当样本变量较多，样本数量不足时\(m>n\), \(\hat \omega^*\)解不唯一

L2正则化

引入对于\(\hat \omega^*\)的L2正则化项

\[\hat J(\hat \omega^*) = (X \hat \omega^* - Y)^T (X\hat \omega^* - Y) + \frac{\lambda}{2} ||\hat \omega^*||_2^2

\]

可以发现，正则化项为\(\hat \omega^*\)二范数平方，乘以\(\frac{\lambda}{2}\)

\(\lambda\)用于控制正则化项和均方误差的权重

\[\left. \frac{{\rm d} \hat J(\hat \omega^*)}{{\rm d} \hat \omega^*} = 0\right.

\]

\[\hat \omega^* = (X^TX + \frac{\lambda}{2}I)^{-1}X^TY

\]

python程序

import numpy as np

import matplotlib.pyplot as plt

from mpl_toolkits.mplot3d import Axes3D

#随机生成两个变量的N个样本

N = 50

feature1 = np.random.rand(N)*10

feature2 = np.random.rand(N)*10

splt = np.ones((1, N))

#

temp_X1 = np.row_stack((feature1, feature2))

temp_X = np.vstack((temp_X1, splt))

X_t = np.mat(temp_X)

X = X_t.T

temp_Y = np.random.rand(N)*10

Y_t = np.mat(temp_Y)

Y = Y_t.T

#画样本散点图

fig = plt.figure()

ax1 = Axes3D(fig)

ax1.scatter(feature1, feature2, temp_Y)

#求Omega

Lbd = 0.01 #确定lambda

I_11 = np.eye(2)

I_12 = np.zeros((2, 1))

I_2 = np.zeros((1, 3))

I_t1 = np.hstack((I_11, I_12))

I_t = np.vstack((I_t1, I_2))

I = np.mat(I_t)

Omega = (X.T*X + Lbd/2*I).I*X.T*Y

#画分回归面

xx = np.linspace(0,10, num=50)

yy = np.linspace(0,10, num=50)

xx_1, yy_1 = np.meshgrid(xx, yy)

Omega_h = np.array(Omega.T)

zz_1 = Omega_h[0, 0]*xx_1 + Omega_h[0, 1]*yy_1 + Omega_h[0, 2]

ax1.plot_surface(xx_1, yy_1, zz_1, alpha= 0.6, color= "r")

plt.show()

效果

参考