参用L2正则化缓解过拟合，区分红蓝点

本文链接：https://blog.csdn.net/wk5032/article/details/132325309

参用L2正则化缓解过拟合，区分红蓝点

欠拟合的解决方法：

增加输入特征

增加网络参数

减少正则化参数

过拟合的解决方法：

数据清洗

增大训练集

采用正则化

增大正则化参数

正则化缓解过拟合

正则化在损失函数中引入模型复杂度指标，利用给W加权值，弱化了训练数据的噪声（一般不正则化b）

loss = loss(y与y_)+REGULARIZER * loss(w)

loss是模型中所有参数的损失函数，如交叉熵，均方误差

REGULARIZER是超参数，给出参数w在总loss中的比例，即正则化的权重

w是需要正则化的参数

L1 正则化

loss $_{L1}$ (w) = $\sum_{i}^{}$ $\left | w_{i} \right |$

L2正则化

loss $_{L2}$ (w) = $\sum_{i}^{}$ $\left | {w_{i}}^{2} \right |$

正则化的选择

L1正则化大概率会使很多的参数变为零，因此该方法可通过稀疏参数，即减少参数的数量，降低复杂度。

L2正则化会使参数很接近零但不为零，因些该方法可通过减少参数值的大小降低复杂度。

没有正则化的代码：

# 导入所需模块
import tensorflow as tf
from matplotlib import pyplot as plt
import numpy as np
import pandas as pd

# 读入数据/标签生成x_train y_train
df = pd.read_csv('dot.csv')
x_data = np.array(df[['x1', 'x2']])
y_data = np.array(df['y_c'])

x_train = np.vstack(x_data).reshape(-1, 2)
y_train = np.vstack(y_data).reshape(-1, 1)

Y_c = [['red' if y else 'blue'] for y in y_train]

# 转换x的数据类型，否则后面矩阵相乘时会因数据类型问题报错
x_train = tf.cast(x_train, tf.float32)
y_train = tf.cast(y_train, tf.float32)

# from_tensor_slices函数切分传入的张量的第一个维度，生成相应的数据集，使输入特征和标签值一一对应
train_db = tf.data.Dataset.from_tensor_slices((x_train, y_train)).batch(32)

# 生成神经网络的参数，输入层为2个神经元，隐藏层为11个神经元，1层隐藏层，输出层为1个神经元
# 用tf.Variable()保证参数可训练
w1 = tf.Variable(tf.random.normal([2, 11]), dtype=tf.float32)
b1 = tf.Variable(tf.constant(0.01, shape=[11]))

w2 = tf.Variable(tf.random.normal([11, 1]), dtype=tf.float32)
b2 = tf.Variable(tf.constant(0.01, shape=[1]))

lr = 0.005 # 学习率
epoch = 800 # 循环轮数

# 训练部分
for epoch in range(epoch):
for step, (x_train, y_train) in enumerate(train_db):
with tf.GradientTape() as tape: # 记录梯度信息

h1 = tf.matmul(x_train, w1) + b1 # 记录神经网络乘加运算
h1 = tf.nn.relu(h1)
y = tf.matmul(h1, w2) + b2

# 采用均方误差损失函数mse = mean(sum(y-out)^2)
loss = tf.reduce_mean(tf.square(y_train - y))

# 计算loss对各个参数的梯度
variables = [w1, b1, w2, b2]
grads = tape.gradient(loss, variables)

# 实现梯度更新
# w1 = w1 - lr * w1_grad tape.gradient是自动求导结果与[w1, b1, w2, b2] 索引为0，1，2，3
w1.assign_sub(lr * grads[0])
b1.assign_sub(lr * grads[1])
w2.assign_sub(lr * grads[2])
b2.assign_sub(lr * grads[3])

# 每20个epoch，打印loss信息
if epoch % 20 == 0:
print('epoch:', epoch, 'loss:', float(loss))

# 预测部分
print("*******predict*******")
# xx在-3到3之间以步长为0.01，yy在-3到3之间以步长0.01,生成间隔数值点
xx, yy = np.mgrid[-3:3:.1, -3:3:.1]
# 将xx , yy拉直，并合并配对为二维张量，生成二维坐标点
grid = np.c_[xx.ravel(), yy.ravel()]
grid = tf.cast(grid, tf.float32)
# 将网格坐标点喂入神经网络，进行预测，probs为输出
probs = []
for x_test in grid:
# 使用训练好的参数进行预测
h1 = tf.matmul([x_test], w1) + b1
h1 = tf.nn.relu(h1)
y = tf.matmul(h1, w2) + b2 # y为预测结果
probs.append(y)

# 取第0列给x1，取第1列给x2
x1 = x_data[:, 0]
x2 = x_data[:, 1]
# probs的shape调整成xx的样子
probs = np.array(probs).reshape(xx.shape)
plt.scatter(x1, x2, color=np.squeeze(Y_c)) # squeeze去掉纬度是1的纬度,相当于去掉[['red'],[''blue]],内层括号变为['red','blue']
# 把坐标xx yy和对应的值probs放入contour函数，给probs值为0.5的所有点上色 plt.show()后显示的是红蓝点的分界线
plt.contour(xx, yy, probs, levels=[.5])
plt.show()

# 读入红蓝点，画出分割线，不包含正则化
# 不清楚的数据，建议print出来查看

epoch: 0 loss: 1.098584771156311
epoch: 20 loss: 0.07028428465127945
epoch: 40 loss: 0.061229243874549866
epoch: 60 loss: 0.05241357907652855
epoch: 80 loss: 0.0474863201379776
epoch: 100 loss: 0.04372150078415871
epoch: 120 loss: 0.0397987887263298
epoch: 140 loss: 0.03709061071276665
epoch: 160 loss: 0.03585193678736687
epoch: 180 loss: 0.035315074026584625
epoch: 200 loss: 0.033282551914453506
epoch: 220 loss: 0.03161897882819176
epoch: 240 loss: 0.030248383060097694
epoch: 260 loss: 0.028747985139489174
epoch: 280 loss: 0.027325252071022987
epoch: 300 loss: 0.025948593392968178
epoch: 320 loss: 0.024906611070036888
epoch: 340 loss: 0.0238435510545969
epoch: 360 loss: 0.02331533096730709
epoch: 380 loss: 0.023085707798600197
epoch: 400 loss: 0.02297861874103546
epoch: 420 loss: 0.022662801668047905
epoch: 440 loss: 0.022246241569519043
epoch: 460 loss: 0.02180095948278904
epoch: 480 loss: 0.021344920620322227
epoch: 500 loss: 0.020856639370322227
epoch: 520 loss: 0.02062300220131874
epoch: 540 loss: 0.020563310012221336
epoch: 560 loss: 0.020480984821915627
epoch: 580 loss: 0.02039160020649433
epoch: 600 loss: 0.02024388127028942
epoch: 620 loss: 0.020062856376171112
epoch: 640 loss: 0.019850777462124825
epoch: 660 loss: 0.019630296155810356
epoch: 680 loss: 0.01943446695804596
epoch: 700 loss: 0.019208915531635284
epoch: 720 loss: 0.019070450216531754
epoch: 740 loss: 0.018948694691061974
epoch: 760 loss: 0.018789947032928467
epoch: 780 loss: 0.018660731613636017
*******predict*******

Process finished with exit code 0

结果轮廓不够平滑存在过拟合现象。

下面参用L2正则化弱化训练数据的噪声

# 导入所需模块
import tensorflow as tf
from matplotlib import pyplot as plt
import numpy as np
import pandas as pd

# 读入数据/标签生成x_train y_train
df = pd.read_csv('dot.csv')
x_data = np.array(df[['x1', 'x2']])
y_data = np.array(df['y_c'])

x_train = x_data
y_train = y_data.reshape(-1, 1)

Y_c = [['red' if y else 'blue'] for y in y_train]

# 转换x的数据类型，否则后面矩阵相乘时会因数据类型问题报错
x_train = tf.cast(x_train, tf.float32)
y_train = tf.cast(y_train, tf.float32)

# from_tensor_slices函数切分传入的张量的第一个维度，生成相应的数据集，使输入特征和标签值一一对应
train_db = tf.data.Dataset.from_tensor_slices((x_train, y_train)).batch(32)

# 生成神经网络的参数，输入层为4个神经元，隐藏层为32个神经元，2层隐藏层，输出层为3个神经元
# 用tf.Variable()保证参数可训练
w1 = tf.Variable(tf.random.normal([2, 11]), dtype=tf.float32)
b1 = tf.Variable(tf.constant(0.01, shape=[11]))

w2 = tf.Variable(tf.random.normal([11, 1]), dtype=tf.float32)
b2 = tf.Variable(tf.constant(0.01, shape=[1]))

lr = 0.005 # 学习率为
epoch = 800 # 循环轮数

# 训练部分
for epoch in range(epoch):
for step, (x_train, y_train) in enumerate(train_db):
with tf.GradientTape() as tape: # 记录梯度信息

h1 = tf.matmul(x_train, w1) + b1 # 记录神经网络乘加运算
h1 = tf.nn.relu(h1)
y = tf.matmul(h1, w2) + b2

# 采用均方误差损失函数mse = mean(sum(y-out)^2)
loss_mse = tf.reduce_mean(tf.square(y_train - y))
# 添加l2正则化
loss_regularization = []
# tf.nn.l2_loss(w)=sum(w ** 2) / 2
loss_regularization.append(tf.nn.l2_loss(w1))
loss_regularization.append(tf.nn.l2_loss(w2))
# 求和
# 例：x=tf.constant(([1,1,1],[1,1,1]))
# tf.reduce_sum(x)
# >>>6
loss_regularization = tf.reduce_sum(loss_regularization)
loss = loss_mse + 0.03 * loss_regularization # REGULARIZER = 0.03

# 计算loss对各个参数的梯度
variables = [w1, b1, w2, b2]
grads = tape.gradient(loss, variables)

# 实现梯度更新
# w1 = w1 - lr * w1_grad
w1.assign_sub(lr * grads[0])
b1.assign_sub(lr * grads[1])
w2.assign_sub(lr * grads[2])
b2.assign_sub(lr * grads[3])

# 每200个epoch，打印loss信息
if epoch % 20 == 0:
print('epoch:', epoch, 'loss:', float(loss))

# 预测部分
print("*******predict*******")
# xx在-3到3之间以步长为0.01，yy在-3到3之间以步长0.01,生成间隔数值点
xx, yy = np.mgrid[-3:3:.1, -3:3:.1]
# 将xx, yy拉直，并合并配对为二维张量，生成二维坐标点
grid = np.c_[xx.ravel(), yy.ravel()]
grid = tf.cast(grid, tf.float32)
# 将网格坐标点喂入神经网络，进行预测，probs为输出
probs = []
for x_predict in grid:
# 使用训练好的参数进行预测
h1 = tf.matmul([x_predict], w1) + b1
h1 = tf.nn.relu(h1)
y = tf.matmul(h1, w2) + b2 # y为预测结果
probs.append(y)

# 取第0列给x1，取第1列给x2
x1 = x_data[:, 0]
x2 = x_data[:, 1]
# probs的shape调整成xx的样子
probs = np.array(probs).reshape(xx.shape)
plt.scatter(x1, x2, color=np.squeeze(Y_c))
# 把坐标xx yy和对应的值probs放入contour函数，给probs值为0.5的所有点上色 plt.show()后显示的是红蓝点的分界线
plt.contour(xx, yy, probs, levels=[.5])
plt.show()

# 读入红蓝点，画出分割线，包含正则化
# 不清楚的数据，建议print出来查看

epoch: 0 loss: 3.4104058742523193
epoch: 20 loss: 0.6932054758071899
epoch: 40 loss: 0.5894945859909058
epoch: 60 loss: 0.5478309392929077
epoch: 80 loss: 0.5137965679168701
epoch: 100 loss: 0.4834403097629547
epoch: 120 loss: 0.455671101808548
epoch: 140 loss: 0.42979568243026733
epoch: 160 loss: 0.4065832793712616
epoch: 180 loss: 0.3849879503250122
epoch: 200 loss: 0.3645938038825989
epoch: 220 loss: 0.3461950719356537
epoch: 240 loss: 0.32906976342201233
epoch: 260 loss: 0.31324759125709534
epoch: 280 loss: 0.29841986298561096
epoch: 300 loss: 0.2845299243927002
epoch: 320 loss: 0.27153727412223816
epoch: 340 loss: 0.2591661214828491
epoch: 360 loss: 0.2476438730955124
epoch: 380 loss: 0.23714624345302582
epoch: 400 loss: 0.22729870676994324
epoch: 420 loss: 0.2180318683385849
epoch: 440 loss: 0.20935100317001343
epoch: 460 loss: 0.2011842429637909
epoch: 480 loss: 0.19350996613502502
epoch: 500 loss: 0.18572543561458588
epoch: 520 loss: 0.178627148270607
epoch: 540 loss: 0.17201372981071472
epoch: 560 loss: 0.16573978960514069
epoch: 580 loss: 0.15980477631092072
epoch: 600 loss: 0.15424701571464539
epoch: 620 loss: 0.1490301489830017
epoch: 640 loss: 0.14396998286247253
epoch: 660 loss: 0.13912735879421234
epoch: 680 loss: 0.134573295712471
epoch: 700 loss: 0.13011564314365387
epoch: 720 loss: 0.12585200369358063
epoch: 740 loss: 0.12190492451190948
epoch: 760 loss: 0.11829013377428055
epoch: 780 loss: 0.11496084183454514
*******predict*******