tf实现loss,优化器，正则项
1. 先导知识点
1.1 tf.where()
条件语句真返回A,条件语句假返回B
tf.where(条件语句，真返回A，假返回B)
代码示例
import tensorflow as tf
a = tf.constant([1, 2, 3, 1, 1])
b = tf.constant([0, 1, 3, 4, 5])
c = tf.where(tf.greater(a, b), a, b) # 若a>b，返回a对应位置的元素，否则返回b对应位置的元素
print("c：", c)
运行结果
c： tf.Tensor([1 2 3 4 5], shape=(5,), dtype=int32)
1.2 np.random.RandomState.rand()
返回一个[0,1)之间的随机数
np.random.RandomState.rand(维度) # 若维度为空，返回标量
代码示例
import numpy as np
rdm = np.random.RandomState(seed=1)
a = rdm.rand()
b = rdm.rand(2, 3)
print("a:", a)
print("b:", b)
运行结果
a: 0.417022004702574
b: [[7.20324493e-01 1.14374817e-04 3.02332573e-01]
[1.46755891e-01 9.23385948e-02 1.86260211e-01]]
1.3 np.vstack()
将两个数组按垂直方向叠加
np.vstack(数组1,数组2)
代码示例：
import numpy as np
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
c = np.vstack((a, b))
print("c:\n", c)
运行结果
c:
[[1 2 3]
[4 5 6]]
1.4 生成网格坐标点
np.mgrid[ ]np.mgrid[起始值:结束值:步长，起始值:结束值:步长，... ]- [起始值，结束值)，区间左闭右开
x.ravel()将x变为一维数组，“把.前变量拉直”np.c_[]使返回的间隔数值点配对np.c_ [数组1，数组2，... ]
代码示例：
import numpy as np
# 生成等间隔数值点
x, y = np.mgrid[1:3:1, 2:4:0.5]
# 将x, y拉直，并合并配对为二维张量，生成二维坐标点
grid = np.c_[x.ravel(), y.ravel()]
print("x:\n", x)
print("y:\n", y)
print("x.ravel():\n", x.ravel())
print("y.ravel():\n", y.ravel())
print('grid:\n', grid)
运行结果
x:
[[1. 1. 1. 1.]
[2. 2. 2. 2.]]
y:
[[2. 2.5 3. 3.5]
[2. 2.5 3. 3.5]]
x.ravel():
[1. 1. 1. 1. 2. 2. 2. 2.]
y.ravel():
[2. 2.5 3. 3.5 2. 2.5 3. 3.5]
grid:
[[1. 2. ]
[1. 2.5]
[1. 3. ]
[1. 3.5]
[2. 2. ]
[2. 2.5]
[2. 3. ]
[2. 3.5]]
np.mgrid[起始值:结束值:步长，起始值:结束值:步长]填入两个值，相当于构建了一个二维坐标，很坐标值为第一个参数，纵坐标值为第二个参数。
例如，横坐标值为[1, 2, 3]，纵坐标为[2, 2.5, 3, 3.5]
x, y = np.mgrid[1:5:1, 2:4:0.5]
print("x:\n", x)
print("y:\n", y)
这样x和y都为3行4列的二维数组，每个点一一对应构成一个二维坐标区域
x:
[[1. 1. 1. 1.]
[2. 2. 2. 2.]
[3. 3. 3. 3.]]
y:
[[2. 2.5 3. 3.5]
[2. 2.5 3. 3.5]
[2. 2.5 3. 3.5]]
2. 复杂度和学习率
2.1 神经网络复杂度
NN复杂度：多用NN层数和NN参数的个数表示
空间复杂度:
层数=隐藏层的层数+ 1个输出层
图中为：2层NN
总参数=总w+总b
第1层：3x4+4
第2层：4x2+2
图中共计：3x4+4 +4x2+2 = 26
时间复杂度:
乘加运算次数
第1层：3x4
第2层：4x2
图中共计：3x4 + 4x2 = 20
2.2 学习率
w t + 1 = w t − l r ∗ ∂ l o s s ∂ w t w_{t+1}=w_{t}-l r * \frac{\partial l o s s}{\partial w_{t}} wt+1=wt−lr∗∂wt∂loss
参数说明
- 更新后的参数
- 当前参数
- 学习率
- 损失函数的梯度（偏导数）
指数衰减学习率
可以先用较大的学习率，快速得到较优解，然后逐步减小学习率，使模型在训练后期稳定。
指数衰减学习率=初始学习率*学习率衰减率(当前轮数1多少轮衰减一次)
代码示例
w = tf.Variable(tf.constant(5, dtype=tf.float32))
epoch = 40
LR_BASE = 0.2 # 最初学习率
LR_DECAY = 0.99 # 学习率衰减率
LR_STEP = 1 # 喂入多少轮BATCH_SIZE后，更新一次学习率
for epoch in range(epoch):
# for epoch 定义顶层循环，表示对数据集循环epoch次，此例数据集数据仅有1个w,初始化时候constant赋值为5，循环40次迭代。
lr = LR_BASE * LR_DECAY ** (epoch / LR_STEP)
with tf.GradientTape() as tape: # with结构到grads框起了梯度的计算过程。
loss = tf.square(w + 1)
grads = tape.gradient(loss, w) # .gradient函数告知谁对谁求导
w.assign_sub(lr * grads)
# .assign_sub 对变量做自减 即：w -= lr*grads 即 w = w - lr*grads
print("After %s epoch,w is %f,loss is %f,lr is %f" % (epoch, w.numpy(), loss, lr))
运行结果，学习率lr在指数衰减
After 0 epoch,w is 2.600000,loss is 36.000000,lr is 0.200000
After 1 epoch,w is 1.174400,loss is 12.959999,lr is 0.198000
After 2 epoch,w is 0.321948,loss is 4.728015,lr is 0.196020
After 3 epoch,w is -0.191126,loss is 1.747547,lr is 0.194060
After 4 epoch,w is -0.501926,loss is 0.654277,lr is 0.192119
After 5 epoch,w is -0.691392,loss is 0.248077,lr is 0.190198
After 6 epoch,w is -0.807611,loss is 0.095239,lr is 0.188296
After 7 epoch,w is -0.879339,loss is 0.037014,lr is 0.186413
After 8 epoch,w is -0.923874,loss is 0.014559,lr is 0.184549
After 9 epoch,w is -0.951691,loss is 0.005795,lr is 0.182703
After 10 epoch,w is -0.969167,loss is 0.002334,lr is 0.180876
After 11 epoch,w is -0.980209,loss is 0.000951,lr is 0.179068
After 12 epoch,w is -0.987226,loss is 0.000392,lr is 0.177277
After 13 epoch,w is -0.991710,loss is 0.000163,lr is 0.175504
After 14 epoch,w is -0.994591,loss is 0.000069,lr is 0.173749
After 15 epoch,w is -0.996452,loss is 0.000029,lr is 0.172012
After 16 epoch,w is -0.997660,loss is 0.000013,lr is 0.170292
After 17 epoch,w is -0.998449,loss is 0.000005,lr is 0.168589
After 18 epoch,w is -0.998967,loss is 0.000002,lr is 0.166903
After 19 epoch,w is -0.999308,loss is 0.000001,lr is 0.165234
After 20 epoch,w is -0.999535,loss is 0.000000,lr is 0.163581
After 21 epoch,w is -0.999685,loss is 0.000000,lr is 0.161946
After 22 epoch,w is -0.999786,loss is 0.000000,lr is 0.160326
After 23 epoch,w is -0.999854,loss is 0.000000,lr is 0.158723
After 24 epoch,w is -0.999900,loss is 0.000000,lr is 0.157136
After 25 epoch,w is -0.999931,loss is 0.000000,lr is 0.155564
After 26 epoch,w is -0.999952,loss is 0.000000,lr is 0.154009
After 27 epoch,w is -0.999967,loss is 0.000000,lr is 0.152469
After 28 epoch,w is -0.999977,loss is 0.000000,lr is 0.150944
After 29 epoch,w is -0.999984,loss is 0.000000,lr is 0.149434
After 30 epoch,w is -0.999989,loss is 0.000000,lr is 0.147940
After 31 epoch,w is -0.999992,loss is 0.000000,lr is 0.146461
After 32 epoch,w is -0.999994,loss is 0.000000,lr is 0.144996
After 33 epoch,w is -0.999996,loss is 0.000000,lr is 0.143546
After 34 epoch,w is -0.999997,loss is 0.000000,lr is 0.142111
After 35 epoch,w is -0.999998,loss is 0.000000,lr is 0.140690
After 36 epoch,w is -0.999999,loss is 0.000000,lr is 0.139283
After 37 epoch,w is -0.999999,loss is 0.000000,lr is 0.137890
After 38 epoch,w is -0.999999,loss is 0.000000,lr is 0.136511
After 39 epoch,w is -0.999999,loss is 0.000000,lr is 0.135146
3. 激活函数
为什么要用激活函数：在神经网络中，如果不对上一层结点的输出做非线性转换的话，再深的网络也是线性模型，只能把输入线性组合再输出，不能学习到复杂的映射关系，因此需要使用激活函数这个非线性函数做转换。
参考：https://www.cnblogs.com/itmorn/p/11132494.html
3.1 Sigmoid函数
sigmod ( x ) = 1 1 + e − x ∈ ( 0 , 1 ) sigmod ′ ( x ) = sigmod ( x ) ∗ ( 1 − sigmod ( x ) ) = 1 1 + e − x ∗ e − x 1 + e − x = e − x ( 1 + e − x ) 2 ∈ ( 0 , 0.25 ) \begin{aligned} &\operatorname{sigmod}(x)=\frac{1}{1+e^{-x}} \in(0,1)\\ &\operatorname{sigmod}^{\prime}(x)=\operatorname{sigmod}(x)^{*}(1-\operatorname{sigmod}(x))=\frac{1}{1+e^{-x}} * \frac{e^{-x}}{1+e^{-x}}=\frac{e^{-x}}{\left(1+e^{-x}\right)^{2}} \in(0,0.25) \end{aligned} sigmod(x)=1+e−x1∈(0,1)sigmod′(x)=sigmod(x)∗(1−sigmod(x))=1+e−x1∗1+e−xe−x=(1+e−x)2e−x∈(0,0.25)
tf.nn.sigmoid(x)
sigmoid函数图像
sigmoid导数图像
目前使用sigmoid函数为激活函数的神经网络已经很少了
特点
(1)易造成梯度消失
深层神经网络更新参数时，需要从输入层到输出层，逐层进行链式求导，而sigmoid函数的导数输出为[0,0.25]间的小数，链式求导需要多层导数连续相乘，这样会出现多个[0,0.25]间的小数连续相乘，从而造成结果趋于0，产生梯度消失，使得参数无法继续更新。
(2)输出非0均值，收敛慢
希望输入每层神经网络的特征是以0为均值的小数值，但sigmoid函数激活后的数据都时整数，使得收敛变慢。
(3)幂运算复杂，训练时间长
sigmoid函数存在幂运算，计算复杂度大。
3.2 Tanh函数
tanh ( x ) = 1 − e − 2 x 1 + e − 2 x ∈ ( − 1 , 1 ) tanh ′ ( x ) = 1 − ( tanh ( x ) ) 2 = 4 e − 2 x ( 1 + e − 2 x ) 2 ∈ ( 0 , 1 ] \begin{array}{l} \tanh (x)=\frac{1-e^{-2 x}}{1+e^{-2 x}} \in(-1,1) \\ \tanh ^{\prime}(x)=1-(\tanh (x))^{2}=\frac{4 e^{-2 x}}{\left(1+e^{-2 x}\right)^{2}} \in(0,1] \end{array} tanh(x)=1+e−2x1−e−2x∈(−1,1)tanh′(x)=1−(tanh(x))2=(1+e−2x)24e−2x∈(0,1]
tf.math.tanh(x)
特点
(1)输出是0均值
(2)易造成梯度消失
(3)幂运算复杂，训练时间长
3.3 Relu函数
r e l u ( x ) = max ( x , 0 ) = { x , x ≥ 0 0 , x < 0 ∈ [ 0 , + ∞ ) r e l u ′ ( x ) = { 1 , x ≥ 0 0 , x < 0 ∈ { 0 , 1 } \begin{array}{l} r e l u(x)=\max (x, 0)=\left\{\begin{array}{l} x, \quad x \geq 0 \\ 0, \quad x<0 \end{array} \in[0,+\infty)\right. \\ r e l u^{\prime}(x)=\left\{\begin{array}{ll} 1, & x \geq 0 \\ 0, & x<0 \end{array} \in\{0,1\}\right. \end{array} relu(x)=max(x,0)={x,x≥00,x<0∈[0,+∞)relu′(x)={1,0,x≥0x<0∈{0,1}
tf.nn.relu(x)
优点:
- 解决了梯度消失问题(在正区间)
- 只 需判断输入是否大于0，计算速度快
- 收敛速度远快于sigmoid和tanh
缺点:
- 输出非0均值，收敛慢
- Dead ReIU问题:某些神经元可能永远不会被激活，导致相应的参数永远不能被更新。
3.4 Leaky Relu函数
LeakyReLU ( x ) = { x , x ≥ 0 a x , x < 0 ∈ R LeakyReL U ′ ( x ) = { 1 , x ≥ 0 a , x < 0 ∈ { a , 1 } \begin{aligned} &\text { LeakyReLU }(x)=\left\{\begin{array}{ll} x, & x \geq 0 \\ a x, & x<0 \end{array} \in R\right.\\ &\text { LeakyReL } U^{\prime}(x)=\left\{\begin{array}{ll} 1, & x \geq 0 \\ a, & x<0 \end{array} \in\{a, 1\}\right. \end{aligned} LeakyReLU (x)={x,ax,x≥0x<0∈R LeakyReL U′(x)={1,a,x≥0x<0∈{a,1}
tf.nn.leaky_relu(x)
理论上来讲，Leaky Relu有Relu的所有优点，外加不会有Dead Relu问题，但是在实际操作当中，并没有完全证明Leaky Relu总是好于Relu。
3.5 总结
- 首选relu激活函数;
- 学习率设置较小值;
- 输入特征标准化，即让输入特征满足以0为均值，1为标准差的正态分布;
- 初始参数中心化，即让随机生成的参数满足以0为均值，下式为标准差的正态分布
2 当前层输入特征个数 \sqrt{\frac{2}{\text { 当前层输入特征个数 }}} 当前层输入特征个数 2
4. 损失函数
损失函数(loss) ：预测值(y) 与已知答案(y_) 的差距
NN优化目标：loss最小，有三种方法
- mse (Mean Squared Error)
- 自定义
- ce (Cross Entropy)
4. 均方误差mes
MSE ( y − , y ) = ∑ i = 1 n ( y − y − ) 2 n \operatorname{MSE}\left(y_{-}, y\right)=\frac{\sum_{i=1}^{n}\left(y-y_{-}\right)^{2}}{n} MSE(y−,y)=n∑i=1n(y−y−)2
loss_mse = tf.reduce_mean(tf.square(y_ - y))
预测酸奶日销量y, x1、 x2是影响日销量的因素。
建模前，应预先采集的数据有:每日x1、x2和销量y_ (即已知答案，最佳情况:产量=销量)
拟造数据集X，Y_ : y_ =x1 + x2，噪声: -0.05~ +0.05
拟合可以预测销量的函数
代码示例
import tensorflow as tf
import numpy as np
SEED = 23455
rdm = np.random.RandomState(seed=SEED) # 生成[0,1)之间的随机数
x = rdm.rand(32, 2)
y_ = [[x1 + x2 + (rdm.rand() / 10.0 - 0.05)] for (x1, x2) in x] # 生成噪声[0,1)/10=[0,0.1); [0,0.1)-0.05=[-0.05,0.05)
x = tf.cast(x, dtype=tf.float32)
w1 = tf.Variable(tf.random.normal([2, 1], stddev=1, seed=1))
epoch = 15000
lr = 0.002
for epoch in range(epoch):
with tf.GradientTape() as tape:
y = tf.matmul(x, w1)
loss_mse = tf.reduce_mean(tf.square(y_ - y))
grads = tape.gradient(loss_mse, w1)
w1.assign_sub(lr * grads)
if epoch % 500 == 0:
print("After %d training steps,w1 is " % (epoch))
print(w1.numpy(), "\n")
print("Final w1 is: ", w1.numpy())
运行结果
......
After 14000 training steps,w1 is
[[0.9993659]
[0.999166 ]]
After 14500 training steps,w1 is
[[1.0002553 ]
[0.99838644]]
Final w1 is: [[1.0009792]
[0.9977485]]
4.2 自定义损失函数
如预测商品销量，预测多了，损失成本;预测少了，损失利润
若利润≠成本，则mse产生的loss无法利益最大化。
自定义损失函数，y_：标准答案数据集的，y：预测答案计算出的
loss
(
y
−
y
)
=
∑
n
f
(
y
，
y
)
\operatorname{loss}\left(y_{-} y\right)=\sum_{n} f\left(y_， y\right)
loss(y−y)=n∑f(y，y)
f ( y − , y ) = { PROFIT* ( y − − y ) y < y − 预测的 y 少了, 损失利高(PROFIT) COST ∗ ( y − y − ) y > = y − 预测的 y 多了，损失成本(COST) f\left(y_{-}, y\right)=\left\{\begin{array}{lll}\text { PROFIT* }\left(y_{-}-y\right) & y<y_{-} & \text {预测的 } y\text { 少了, 损失利高(PROFIT) } \\ \text { COST } *\left(y-y_{-}\right) & y>=y_{-} & \text {预测的 } y \text { 多了，损失成本(COST) }\end{array}\right. f(y−,y)={ PROFIT* (y−−y) COST ∗(y−y−)y<y−y>=y−预测的 y 少了, 损失利高(PROFIT) 预测的 y 多了，损失成本(COST)
如:预测酸奶销量，酸奶成本(COST) 1元，酸奶利润(PROFIT) 99元。
预测少了损失利润99元，大于预测多了损失成本1元。预测少了损失大，希望生成的预测函数往多了预测。
则损失函数为
loss = tf.reduce_sum(tf.where(tf.greater(y, y_), (y - y_) * COST, (y_ - y) * PROFIT))
代码示例
import tensorflow as tf
import numpy as np
SEED = 23455
COST = 1
PROFIT = 99
rdm = np.random.RandomState(SEED)
x = rdm.rand(32, 2)
y_ = [[x1 + x2 + (rdm.rand() / 10.0 - 0.05)] for (x1, x2) in x] # 生成噪声[0,1)/10=[0,0.1); [0,0.1)-0.05=[-0.05,0.05)
x = tf.cast(x, dtype=tf.float32)
w1 = tf.Variable(tf.random.normal([2, 1], stddev=1, seed=1))
epoch = 10000
lr = 0.002
for epoch in range(epoch):
with tf.GradientTape() as tape:
y = tf.matmul(x, w1)
loss = tf.reduce_sum(tf.where(tf.greater(y, y_), (y - y_) * COST, (y_ - y) * PROFIT))
grads = tape.gradient(loss, w1)
w1.assign_sub(lr * grads)
if epoch % 500 == 0:
print("After %d training steps,w1 is " % (epoch))
print(w1.numpy(), "\n")
print("Final w1 is: ", w1.numpy())
自定义损失函数，酸奶成本1元， 酸奶利润99元，成本很低，利润很高，人们希望多预测些，生成模型系数大于1，往多了预测。运行结果
......
After 9000 training steps,w1 is
[[1.1707956]
[1.0338644]]
After 9500 training steps,w1 is
[[1.1611756]
[1.0651482]]
Final w1 is: [[1.1626335]
[1.1191947]]
自定义损失函数，酸奶成本99元， 酸奶利润1元，成本很高，利润很低，人们希望多少预测，生成模型系数小于1，往少了预测。运行结果
......
After 9000 training steps,w1 is
[[0.8042901]
[0.7632218]]
After 9500 training steps,w1 is
[[0.8998995 ]
[0.82509875]]
Final w1 is: [[0.9205433]
[0.9186459]]
4.3 交叉熵损失函数
交义熵损失函数CE (Cross Entropy)：表征两个概率分布之间的距离
H
(
y
−
,
y
)
=
−
∑
y
−
∗
ln
y
\mathrm{H}\left(\mathrm{y}_{-}, \mathrm{y}\right)=-\sum y_{-} * \ln y
H(y−,y)=−∑y−∗lny
eg.二分类已知答案y_ =(1, 0)，预测y1=(0.6, 0.4) y2=(0.8, 0.2)哪个更接近标准答案?
H
1
(
(
1
,
0
)
,
(
0.6
,
0.4
)
)
=
−
(
1
∗
ln
0.6
+
0
∗
ln
0.4
)
≈
−
(
−
0.511
+
0
)
=
0.511
H
2
(
(
1
,
0
)
,
(
0.8
,
0.2
)
)
=
−
(
1
∗
ln
0.8
+
0
∗
ln
0.2
)
≈
−
(
−
0.223
+
0
)
=
0.223
\begin{aligned} &\mathrm{H}_{1}((1,0),(0.6,0.4))=-(1 * \ln 0.6+0 * \ln 0.4) \approx-(-0.511+0)=0.511\\ &\mathrm{H}_{2}((1,0),(0.8,0.2))=-(1 * \ln 0.8+0 * \ln 0.2) \approx-(-0.223+0)=0.223 \end{aligned}
H1((1,0),(0.6,0.4))=−(1∗ln0.6+0∗ln0.4)≈−(−0.511+0)=0.511H2((1,0),(0.8,0.2))=−(1∗ln0.8+0∗ln0.2)≈−(−0.223+0)=0.223
因为H> H2，所以y2预测更准
tf.losses.categorical crossentropy(y_ ，y)
代码示例
loss_ce1 = tf.losses.categorical_crossentropy([1, 0], [0.6, 0.4])
loss_ce2 = tf.losses.categorical_crossentropy([1, 0], [0.8, 0.2])
print("loss_ce1:", loss_ce1)
print("loss_ce2:", loss_ce2)
# 交叉熵损失函数
运行结果
loss_ce1: tf.Tensor(0.5108256, shape=(), dtype=float32)
loss_ce2: tf.Tensor(0.22314353, shape=(), dtype=float32)
4.4 交叉熵损失函数与softmax结合
输出先过softmax函数，再计算y与y_ 的交叉熵损失函数。
tf.nn.softmax_cross_entropy_with_logits(y_, y)
代码示例
# softmax与交叉熵损失函数的结合
import tensorflow as tf
import numpy as np
y_ = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1], [1, 0, 0], [0, 1, 0]])
y = np.array([[12, 3, 2], [3, 10, 1], [1, 2, 5], [4, 6.5, 1.2], [3, 6, 1]])
y_pro = tf.nn.softmax(y)
loss_ce1 = tf.losses.categorical_crossentropy(y_,y_pro)
loss_ce2 = tf.nn.softmax_cross_entropy_with_logits(y_, y)
print('分步计算的结果:\n', loss_ce1)
print('结合计算的结果:\n', loss_ce2)
# 输出的结果相同
运行结果
分步计算的结果:
tf.Tensor(
[1.68795487e-04 1.03475622e-03 6.58839038e-02 2.58349207e+00
5.49852354e-02], shape=(5,), dtype=float64)
结合计算的结果:
tf.Tensor(
[1.68795487e-04 1.03475622e-03 6.58839038e-02 2.58349207e+00
5.49852354e-02], shape=(5,), dtype=float64)
5. 过拟合与欠拟合
欠拟合的解决方法:
- 增加输入特征项
- 增加网络参数
- 减少正则化参数
过拟合的解决方法:
- 数据清洗
- 增大训练集
- 采用正则化
- 增大正则化参数
5.1 正则化缓解过拟合
正则化在损失函数中引入模型复杂度指标，利用给W加权值，弱化了训练数据的噪声(一般不正则化b)
loss
=
loss
(
y
与
y
−
)
+
R
E
G
U
L
A
R
I
Z
E
R
∗
loss
(
w
)
\operatorname{loss}=\operatorname{loss}\left(\mathrm{y}与{y}_{-}\right)+\mathrm{REGULARIZER}{*} \operatorname{loss}(\mathrm{w})
loss=loss(y与y−)+REGULARIZER∗loss(w)
式中含义：
loss(y与y_)：模型中所有参数的损失函数。如:交叉熵、均方误差
REGULARIZER：用超参数REGULARIZER给出参数w在总loss中的比例，即正则化的权重
loss(w)：需要正则化的参数。计算方式有两种
loss
L
1
(
w
)
=
∑
i
∣
w
i
∣
\operatorname{loss}_{L_{1}}(w)=\sum_{i}\left|w_{i}\right|
lossL1(w)=i∑∣wi∣
loss L 2 ( w ) = ∑ i ∣ w i 2 ∣ \operatorname{loss}_{L 2}(w)=\sum_{i}\left|w_{i}^{2}\right| lossL2(w)=i∑∣∣wi2∣∣
正则化的选择
- L1正则化大概率会使很多参数变为零，因此该方法可通过稀疏参数，即减少参数的数量，降低复杂度。
- L2正则化会使参数很接近零但不为零，因此该方法可通过减小参数值的大小降低复杂度。
tf.nn.l2_loss(w)
代码示例，未采用正则化p29_regularizationfree.py
# 导入所需模块
import tensorflow as tf
from matplotlib import pyplot as plt
import numpy as np
import pandas as pd
# 读入数据/标签 生成x_train y_train
df = pd.read_csv('dot.csv')
x_data = np.array(df[['x1', 'x2']])
y_data = np.array(df['y_c'])
# reshape(-1,x) -1是将一维数组转换为二维的矩阵，并且第二个参数是表示分成几列，
# 但是在reshape的时候必须让数组里面的个数和shape的函数做取余时值为零才能转换
x_train = np.vstack(x_data).reshape(-1,2)
y_train = np.vstack(y_data).reshape(-1,1) #将y_data转换为二维数组
Y_c = [['red' if y else 'blue'] for y in y_train] # 三元运算
# 转换x的数据类型，否则后面矩阵相乘时会因数据类型问题报错
x_train = tf.cast(x_train, tf.float32)
y_train = tf.cast(y_train, tf.float32)
# from_tensor_slices函数切分传入的张量的第一个维度，生成相应的数据集，使输入特征和标签值一一对应
train_db = tf.data.Dataset.from_tensor_slices((x_train, y_train)).batch(32)
# 生成神经网络的参数，输入层为2个神经元，隐藏层为11个神经元，1层隐藏层，输出层为1个神经元
# 隐藏层11个神经元为人为指定
# 用tf.Variable()保证参数可训练
w1 = tf.Variable(tf.random.normal([2, 11]), dtype=tf.float32) # 隐藏层2个输入，11个输出
b1 = tf.Variable(tf.constant(0.01, shape=[11])) # b的个数与w个数相同
w2 = tf.Variable(tf.random.normal([11, 1]), dtype=tf.float32) # 输出层接收11个，输出1个
b2 = tf.Variable(tf.constant(0.01, shape=[1]))
lr = 0.01 # 学习率
epoch = 400 # 循环轮数
# 训练部分
for epoch in range(epoch):
for step, (x_train, y_train) in enumerate(train_db):
with tf.GradientTape() as tape: # 记录梯度信息
h1 = tf.matmul(x_train, w1) + b1 # 记录神经网络乘加运算
h1 = tf.nn.relu(h1) # relu激活函数
y = tf.matmul(h1, w2) + b2
# 采用均方误差损失函数mse = mean(sum(y-out)^2)
loss = tf.reduce_mean(tf.square(y_train - y))
# 计算loss对各个参数的梯度
variables = [w1, b1, w2, b2]
grads = tape.gradient(loss, variables)
# 实现梯度更新
# w1 = w1 - lr * w1_grad tape.gradient是自动求导结果与[w1, b1, w2, b2] 索引为0，1，2，3
w1.assign_sub(lr * grads[0])
b1.assign_sub(lr * grads[1])
w2.assign_sub(lr * grads[2])
b2.assign_sub(lr * grads[3])
# 每20个epoch，打印loss信息
if epoch % 20 == 0:
print('epoch:', epoch, 'loss:', float(loss))
# 预测部分
print("*******predict*******")
# xx在-3到3之间以步长为0.01，yy在-3到3之间以步长0.01,生成间隔数值点
xx, yy = np.mgrid[-3:3:.1, -3:3:.1]
# 将xx , yy拉直，并合并配对为二维张量，生成二维坐标点
grid = np.c_[xx.ravel(), yy.ravel()]
grid = tf.cast(grid, tf.float32)
# 将网格坐标点喂入神经网络，进行预测，probs为输出
probs = []
for x_test in grid:
# 使用训练好的参数进行预测
h1 = tf.matmul([x_test], w1) + b1
h1 = tf.nn.relu(h1)
y = tf.matmul(h1, w2) + b2 # y为预测结果
probs.append(y)
# 取第0列给x1，取第1列给x2
x1 = x_data[:, 0]
x2 = x_data[:, 1]
# probs的shape调整成xx的样子
probs = np.array(probs).reshape(xx.shape)
plt.scatter(x1, x2, color=np.squeeze(Y_c)) # squeeze去掉纬度是1的纬度,相当于去掉[['red'],['blue']],内层括号变为['red','blue']
# 把坐标xx yy和对应的值probs放入contour<[‘kɑntʊr]>函数，给probs值为0.5的所有点上色 plt点show后 显示的是红蓝点的分界线
plt.contour(xx, yy, probs, levels=[.5]) # 画出probs值为0.5轮廓线,levels:这个参数用于显示具体哪几条登高线
plt.show()
# 读入红蓝点，画出分割线，不包含正则化
# 不清楚的数据，建议print出来查看
运行结果
epoch: 0 loss: 1.6901788711547852
epoch: 20 loss: 0.06456395983695984
epoch: 40 loss: 0.0639718547463417
epoch: 60 loss: 0.054891664534807205
epoch: 80 loss: 0.037164993584156036
epoch: 100 loss: 0.0290686022490263
epoch: 120 loss: 0.026631897315382957
epoch: 140 loss: 0.025654718279838562
epoch: 160 loss: 0.025450214743614197
epoch: 180 loss: 0.02445397339761257
epoch: 200 loss: 0.02315516769886017
epoch: 220 loss: 0.02262507937848568
epoch: 240 loss: 0.02210732363164425
epoch: 260 loss: 0.02202308177947998
epoch: 280 loss: 0.022013641893863678
epoch: 300 loss: 0.02216213382780552
epoch: 320 loss: 0.02226211130619049
epoch: 340 loss: 0.022413412109017372
epoch: 360 loss: 0.022659024223685265
epoch: 380 loss: 0.02281317301094532
*******predict*******
代码示例，在训练部分采用L2正则化
# 导入所需模块
import tensorflow as tf
from matplotlib import pyplot as plt
import numpy as np
import pandas as pd
# 读入数据/标签 生成x_train y_train
df = pd.read_csv('dot.csv')
x_data = np.array(df[['x1', 'x2']])
y_data = np.array(df['y_c'])
x_train = x_data
y_train = y_data.reshape(-1, 1)
Y_c = [['red' if y else 'blue'] for y in y_train]
# 转换x的数据类型，否则后面矩阵相乘时会因数据类型问题报错
x_train = tf.cast(x_train, tf.float32)
y_train = tf.cast(y_train, tf.float32)
# from_tensor_slices函数切分传入的张量的第一个维度，生成相应的数据集，使输入特征和标签值一一对应
train_db = tf.data.Dataset.from_tensor_slices((x_train, y_train)).batch(32)
# 生成神经网络的参数，输入层为4个神经元，隐藏层为32个神经元，2层隐藏层，输出层为3个神经元
# 用tf.Variable()保证参数可训练
w1 = tf.Variable(tf.random.normal([2, 11]), dtype=tf.float32)
b1 = tf.Variable(tf.constant(0.01, shape=[11]))
w2 = tf.Variable(tf.random.normal([11, 1]), dtype=tf.float32)
b2 = tf.Variable(tf.constant(0.01, shape=[1]))
lr = 0.01 # 学习率为
epoch = 400 # 循环轮数
# 训练部分
for epoch in range(epoch):
for step, (x_train, y_train) in enumerate(train_db):
with tf.GradientTape() as tape: # 记录梯度信息
h1 = tf.matmul(x_train, w1) + b1 # 记录神经网络乘加运算
h1 = tf.nn.relu(h1)
y = tf.matmul(h1, w2) + b2
# 采用均方误差损失函数mse = mean(sum(y-out)^2)
loss_mse = tf.reduce_mean(tf.square(y_train - y))
# 添加l2正则化
loss_regularization = []
# tf.nn.l2_loss(w)=sum(w ** 2) / 2
loss_regularization.append(tf.nn.l2_loss(w1))
loss_regularization.append(tf.nn.l2_loss(w2))
# 求和
# 例：x=tf.constant(([1,1,1],[1,1,1]))
# tf.reduce_sum(x)
# >>>6
# loss_regularization = tf.reduce_sum(tf.stack(loss_regularization))
loss_regularization = tf.reduce_sum(loss_regularization)
loss = loss_mse + 0.03 * loss_regularization # REGULARIZER = 0.03
# 计算loss对各个参数的梯度
variables = [w1, b1, w2, b2]
grads = tape.gradient(loss, variables)
# 实现梯度更新
# w1 = w1 - lr * w1_grad
w1.assign_sub(lr * grads[0])
b1.assign_sub(lr * grads[1])
w2.assign_sub(lr * grads[2])
b2.assign_sub(lr * grads[3])
# 每200个epoch，打印loss信息
if epoch % 20 == 0:
print('epoch:', epoch, 'loss:', float(loss))
# 预测部分
print("*******predict*******")
# xx在-3到3之间以步长为0.01，yy在-3到3之间以步长0.01,生成间隔数值点
xx, yy = np.mgrid[-3:3:.1, -3:3:.1]
# 将xx, yy拉直，并合并配对为二维张量，生成二维坐标点
grid = np.c_[xx.ravel(), yy.ravel()]
grid = tf.cast(grid, tf.float32)
# 将网格坐标点喂入神经网络，进行预测，probs为输出
probs = []
for x_predict in grid:
# 使用训练好的参数进行预测
h1 = tf.matmul([x_predict], w1) + b1
h1 = tf.nn.relu(h1)
y = tf.matmul(h1, w2) + b2 # y为预测结果
probs.append(y)
# 取第0列给x1，取第1列给x2
x1 = x_data[:, 0]
x2 = x_data[:, 1]
# probs的shape调整成xx的样子
probs = np.array(probs).reshape(xx.shape)
plt.scatter(x1, x2, color=np.squeeze(Y_c))
# 把坐标xx yy和对应的值probs放入contour<[‘kɑntʊr]>函数，给probs值为0.5的所有点上色 plt点show后 显示的是红蓝点的分界线
plt.contour(xx, yy, probs, levels=[.5])
plt.show()
# 读入红蓝点，画出分割线，包含正则化
# 不清楚的数据，建议print出来查看
运行结果
epoch: 0 loss: 1.530280351638794
epoch: 20 loss: 0.7782743573188782
epoch: 40 loss: 0.6781619191169739
epoch: 60 loss: 0.5953636765480042
epoch: 80 loss: 0.5263288617134094
epoch: 100 loss: 0.4674427807331085
epoch: 120 loss: 0.41659849882125854
epoch: 140 loss: 0.37269479036331177
epoch: 160 loss: 0.3337797522544861
epoch: 180 loss: 0.3002385199069977
epoch: 200 loss: 0.27038004994392395
epoch: 220 loss: 0.24350212514400482
epoch: 240 loss: 0.22041508555412292
epoch: 260 loss: 0.20032131671905518
epoch: 280 loss: 0.1829461306333542
epoch: 300 loss: 0.16758175194263458
epoch: 320 loss: 0.15422624349594116
epoch: 340 loss: 0.14259028434753418
epoch: 360 loss: 0.13238896429538727
epoch: 380 loss: 0.12349270284175873
*******predict*******
补充内容：python matplotlib contour画等高线图
https://blog.csdn.net/Mr_Cat123/article/details/80677525
6. 神经网络参数优化器
优化器：是引导神经网络更新参数的工具
作用：用来更新和计算影响模型训练和模型输出的网络参数，使其逼近或达到最优值，从而最小化(或最大化)损失函数
待优化参数w，损失函数loss, 学习率lr， 每次迭代个batch（每个batch包含2^n组数据），t表示当前batch迭代的总次数:
1.计算t时刻损失函数关于当前参数的梯度
g
t
=
∇
loss
=
∂
loss
∂
(
w
t
)
g_{t}=\nabla \operatorname{loss}=\frac{\partial \operatorname{loss}}{\partial\left(w_{t}\right)}
gt=∇loss=∂(wt)∂loss
2.计算t时刻一阶动量mt和二阶动量Vt
- 一阶动量:与梯度相关的函数
- 二阶动量:与梯度平方相关的函数
3.计算t时刻下降梯度:
η
t
=
l
r
⋅
m
t
/
V
t
\eta_{\mathrm{t}}=l r \cdot m_{\mathrm{t}} / \sqrt{V_{\mathrm{t}}}
ηt=lr⋅mt/Vt
4.计算t+1时刻参数
w
t
+
1
=
w
t
−
η
t
=
w
t
−
l
r
⋅
m
t
/
V
t
w_{\mathrm{t}+1}=w_{t}-\eta_{t}=w_{t}-l r \cdot m_{t} / \sqrt{V_{t}}
wt+1=wt−ηt=wt−lr⋅mt/Vt
不同的优化器实质上只是定义了不同的一阶动量和二阶动量公式
6.1 SGD 随机梯度下降
SGD (无momentum)，常用的梯度下降法。
m
t
=
g
t
V
t
=
1
m_{\mathrm{t}}=g_{\mathrm{t}} \quad V_{\mathrm{t}}=1
mt=gtVt=1
η t = l r ⋅ m t / V t = l r ⋅ g t \boldsymbol{\eta}_{\mathrm{t}}=\boldsymbol{l} \boldsymbol{r} \cdot \boldsymbol{m}_{\mathrm{t}} / \sqrt{\boldsymbol{V}_{t}}=\boldsymbol{l} \boldsymbol{r} \cdot \boldsymbol{g}_{t} ηt=lr⋅mt/Vt=lr⋅gt
w t + 1 = w t − η t = w t − l r ⋅ m t / v t = w t − l r ⋅ g t \begin{aligned} w_{t+1}=& w_{t}-\eta_{t} \\ &=w_{t}-l r \cdot m_{t} / \sqrt{v_{t}}=w_{t}-lr \cdot g_{t} \end{aligned} wt+1=wt−ηt=wt−lr⋅mt/vt=wt−lr⋅gt
即为
w
t
+
1
=
w
t
−
l
r
∗
∂
l
o
s
s
∂
w
t
\mathrm{w}_{\mathrm{t}+1}=w_{t}-l r * \frac{\partial l o s s}{\partial w_{t}}
wt+1=wt−lr∗∂wt∂loss
6.2 SGDM
( SGDM (含momentum的SGD)，在SGD基础上增加一 阶动量。
m
t
=
β
⋅
m
t
−
1
+
(
1
−
β
)
⋅
g
t
m_{\mathrm{t}}=\beta \cdot m_{t-1}+(1-\beta) \cdot g_{t}
mt=β⋅mt−1+(1−β)⋅gt
mt：表示各时刻梯度方向的指数滑动平均值
β：超参数，趋近于1，经验值为0.9
V
t
=
1
V_{\mathrm{t}}=1
Vt=1
η t = lr ⋅ m t / V t = lr ⋅ m t = lr ⋅ ( β ⋅ m t − 1 + ( 1 − β ) ⋅ g t ) \begin{aligned} \eta_{\mathrm{t}}=& \operatorname{lr} \cdot m_{\mathrm{t}} / \sqrt{V_{\mathrm{t}}}=\operatorname{lr} \cdot m_{\mathrm{t}} \\ &=\operatorname{lr} \cdot\left(\beta \cdot m_{\mathrm{t}-1}+(1-\beta) \cdot g_{\mathrm{t}}\right) \end{aligned} ηt=lr⋅mt/Vt=lr⋅mt=lr⋅(β⋅mt−1+(1−β)⋅gt)
w t + 1 = w t − η t = w t − l r ⋅ ( β ⋅ m t − 1 + ( 1 − β ) ⋅ g t ) \begin{aligned} w_{\mathrm{t}+1}=& w_{\mathrm{t}}-\eta_{\mathrm{t}} \\ &=w_{\mathrm{t}}-l r \cdot\left(\beta \cdot m_{\mathrm{t}-1}+(1-\beta) \cdot g_{\mathrm{t}}\right) \end{aligned} wt+1=wt−ηt=wt−lr⋅(β⋅mt−1+(1−β)⋅gt)
6.3 Adagrad
Adagrad, 在SGD基础上增加二阶动量
m
t
=
g
t
m_{\mathrm{t}}=g_{\mathrm{t}}
mt=gt
二阶动量是从开始到现在梯度平方的累计和:
V
t
=
∑
τ
=
1
t
g
τ
2
V_{t}=\sum_{\tau=1}^{t} g_{\tau}^{2}
Vt=τ=1∑tgτ2
η t = l r ⋅ m t / ( V t ) = l r ⋅ g t / ( ∑ τ = 1 t g t 2 ) \begin{array}{l} \eta_{\mathrm{t}}=lr \cdot m_{t} /(\sqrt{V_{t}}) \\ \quad=lr \cdot g_{t} /(\sqrt{\sum_{\tau=1}^{t} g_{t}^{2}}) \end{array} ηt=lr⋅mt/(Vt)=lr⋅gt/(∑τ=1tgt2)
w t + 1 = w t − η t = w t − l r ⋅ g t / ( ∑ τ = 1 t g t 2 ) \begin{aligned} w_{t+1}=& w_{t}-\eta_{t} \\ &=w_{t}-lr \cdot g_{t} /(\sqrt{\sum_{\tau=1}^{t} g_{t}^{2}}) \end{aligned} wt+1=wt−ηt=wt−lr⋅gt/(τ=1∑tgt2)
6.4 RMSProp
RMSProp, SGD基础上增加二 阶动量
m
t
=
g
t
m_{\mathrm{t}}=g_{\mathrm{t}}
mt=gt
二阶动量v使用指数滑动平均值计算，表征的是过去一段时间的平均值
V
t
=
β
⋅
V
t
−
1
+
(
1
−
β
)
⋅
g
t
2
V_{t}=\beta \cdot V_{t-1}+(1-\beta) \cdot g_{t}^{2}
Vt=β⋅Vt−1+(1−β)⋅gt2
η t = l r ⋅ m t / V t = l r ⋅ g t / ( β ⋅ V t − 1 + ( 1 − β ) ⋅ g t 2 ) \begin{aligned} \eta_{t}=& l r \cdot m_{\mathrm{t}} / \sqrt{V_{\mathrm{t}}} \\ &=lr \cdot g_{t} /(\sqrt{\beta \cdot V_{t-1}+(1-\beta) \cdot g_{t}^{2}}) \end{aligned} ηt=lr⋅mt/Vt=lr⋅gt/(β⋅Vt−1+(1−β)⋅gt2)
w t + 1 = w t − η t = w t − l r ⋅ g t / ( β ⋅ V t − 1 + ( 1 − β ) ⋅ g t 2 ) \begin{aligned} w_{t+1} &=w_{t}-\eta_{t} \\ &=w_{t}-lr \cdot g_{t} /(\sqrt{\beta \cdot V_{t-1}+(1-\beta) \cdot g_{t}^{2}}) \end{aligned} wt+1=wt−ηt=wt−lr⋅gt/(β⋅Vt−1+(1−β)⋅gt2)
6.5 Adam
Adam，同时结合SGDM一阶动量和RMSProp二阶动量
一阶动量：
m
t
=
β
1
⋅
m
t
−
1
+
(
1
−
β
1
)
⋅
g
t
m_{\mathrm{t}}=\beta_{1} \cdot m_{t-1}+\left(1-\beta_{1}\right) \cdot g_{t}
mt=β1⋅mt−1+(1−β1)⋅gt
修正一阶动量的偏差，t为从训练开始到当前时刻所经历的总batch数::
m
^
t
=
m
t
1
−
β
1
t
\widehat{m}_{\mathrm{t}}=\frac{m_{\mathrm{t}}}{1-\beta_{1}^{t}}
m
t=1−β1tmt
二阶动量：
V
t
=
β
2
⋅
V
s
t
e
p
−
1
+
(
1
−
β
2
)
⋅
g
t
2
V_{t}=\beta_{2} \cdot V_{s t e p-1}+\left(1-\beta_{2}\right) \cdot g_{t}^{2}
Vt=β2⋅Vstep−1+(1−β2)⋅gt2
修正二阶动量的偏差，t为从训练开始到当前时刻所经历的总batch数:
V
t
^
=
V
t
1
−
β
2
t
\widehat{V_{t}}=\frac{V_{t}}{1-\beta_{2}^{t}}
Vt
=1−β2tVt
η t = l r ⋅ m ^ t / V ^ t = lr ⋅ m t 1 − β 1 t / V t 1 − β 2 t \begin{aligned} \eta_{t}=& lr \cdot \widehat{m}_{\mathrm{t}} / \sqrt{\widehat{V}_{t}} \\ &=\operatorname{lr} \cdot \frac{m_{\mathrm{t}}}{1-\beta_{1}^{t}} / \sqrt{\frac{V_{t}}{1-\beta_{2}^{t}}} \end{aligned} ηt=lr⋅m t/V t=lr⋅1−β1tmt/1−β2tVt
w t + 1 = w t − η t = w t − l r ⋅ m t 1 − β 1 t / v t 1 − β 2 t \begin{aligned} w_{t+1} &=w_{t}-\eta_{t} \\ &=w_{t}-l r \cdot \frac{m_{t}}{1-\beta_{1}^{t}} / \sqrt{\frac{v_{t}}{1-\beta_{2}^{t}}} \end{aligned} wt+1=wt−ηt=wt−lr⋅1−β1tmt/1−β2tvt
6.6 优化器对比
6.6.1 SGD
loss图像
acc图像
耗时：12.678699254989624
6.6.2 SGDM
loss图像
acc图像
耗时：17.32265305519104
6.6.3 Adagrad
loss图像
acc图像
耗时：13.080469131469727
6.6.4 RMSProp
loss图像
acc图像
耗时：16.42955780029297
6.6.5 Adam
loss图像
acc图像
耗时：22.04225492477417
- 参考：北大tensorflow2.0教程
736
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
