PyTorch学习-梯度下降算法

分治方法：先分成n×n个点进行搜索，选择最低的点，对最低的点再分成n×n份再次进行搜索，选择最小的值，反复如此，找到最小值。但是这对于凸函数是较为有用的，对于不规则的函数，可能搜索到的是极小值点（局部最优点），而不是最小值点（全局最优点）。

Gradient Descent Algorithm : 梯度下降算法（贪心思想，局部最优）
Gradient : 梯度，梯度大于0上升，梯度小于0下降，所以参数向梯度的反方向更新。
$w=w-x{g}^{\prime }(w)$
x：学习率

损失函数的局部最优点比较少，但是我们有可能遇到鞍点，鞍点的导数等于0（梯度等于0, $g^{\prime }(w)=0$），这时候参数无法更新。

对损失函数求导，求参数更新公式：

根据公式写出代码：

import numpy as np
import matplotlib.pyplot as plt
x_data = [1.0, 2.0, 3.0]
y_data = [2.0, 4.0, 6.0]

w = 1.0  # y = x * w， w的初始值

def forward(x):
    return x * w
    
def cost(xs, xy):
    cost = 0
    for x, y in zip(xs, xy):
        y_pred = forward(x)
        cost += (y_pred - y) ** 2
    return cost / len(xs)

def gradient(xs, ys):
    grad = 0
    for x, y in zip(xs, ys):
        grad += 2 * x * (x * w - y)
    return grad / len(xs)

def gradient(xs, ys):
    grad = 0
    for x, y in zip(xs, ys):
        grad += 2 * x * (x * w - y)
    return grad / len(xs)

loss_list = []
trainnum_list = []
print("Predict (before training)", 4, forward(4)) # 训练前x = 4，对应预测的y值
for epoch in range(100):
    cost_val = cost(x_data, y_data)
    grad_val = gradient(x_data, y_data)
    w -= 0.01 * grad_val
    print("Epoch: ", epoch, "w = ", w, "loss = ", cost_val)
    loss_list.append(cost_val)
    trainnum_list.append(epoch)
print("Predict(after training)", 4, forward(4))  # 训练后x = 4，对应预测的y值

输出：

Predict (before training) 4 4.0
Epoch: 0 w = 1.0933333333333333 loss = 4.666666666666667
Epoch: 1 w = 1.1779555555555554 loss = 3.8362074074074086
Epoch: 2 w = 1.2546797037037036 loss = 3.1535329869958857
Epoch: 3 w = 1.3242429313580246 loss = 2.592344272332262
Epoch: 4 w = 1.3873135910979424 loss = 2.1310222071581117
Epoch: 5 w = 1.4444976559288012 loss = 1.7517949663820642
Epoch: 6 w = 1.4963445413754464 loss = 1.440053319920117
Epoch: 7 w = 1.5433523841804047 loss = 1.1837878313441108
Epoch: 8 w = 1.5859728283235668 loss = 0.9731262101573632
Epoch: 9 w = 1.6246153643467005 loss = 0.7999529948031382
Epoch: 10 w = 1.659651263674342 loss = 0.6575969151946154
Epoch: 11 w = 1.6914171457314033 loss = 0.5405738908195378
Epoch: 12 w = 1.7202182121298057 loss = 0.44437576375991855
··· ···
Epoch: 92 w = 1.9998896858085284 loss = 6.908348768398496e-08
Epoch: 93 w = 1.9998999817997325 loss = 5.678969725349543e-08
Epoch: 94 w = 1.9999093168317574 loss = 4.66836551287917e-08
Epoch: 95 w = 1.9999177805941268 loss = 3.8376039345125727e-08
Epoch: 96 w = 1.9999254544053418 loss = 3.154680994333735e-08
Epoch: 97 w = 1.9999324119941766 loss = 2.593287985380858e-08
Epoch: 98 w = 1.9999387202080534 loss = 2.131797981222471e-08
Epoch: 99 w = 1.9999444396553017 loss = 1.752432687141379e-08
Predict(after training) 4 7.999777758621207

画出loss的变化趋势：

plt.plot(trainnum_list, loss_list)
plt.ylabel("Cost")
plt.xlabel("Epoch")
plt.show()

图像：

Stochastic Gradient Descent 随机梯度下降

使用单个数据求梯度，因为即使陷入鞍点，随机噪声也会让我们越过鞍点继续更新。

参数更新公式：
$w=w-\alpha \frac{\partial loss}{\partial w}=w-\alpha \cdot [2\cdot x_n\cdot (x_n\cdot w-y_n)]$

修改之前的代码：

import numpy as np
import matplotlib.pyplot as plt
x_data = [1.0, 2.0, 3.0]
y_data = [2.0, 4.0, 6.0]

w = 1.0  # y = x * w， w的初始值

def forward(x):
    return x * w
    
def loss(x, y):
    y_pred = forward(x)
    return (y_pred - y) ** 2

def gradient(x, y):
    return 2 * x * (x * w - y)

loss_list = []
trainnum_list = []
print("Predict (before training)", 4, forward(4)) # 训练前x = 4，对应预测的y值
for epoch in range(100):
    for x, y in zip(x_data, y_data):
        grad = gradient(x, y)
        w -= 0.01 * grad
        print("\tgrad", x, y, grad)
        l = loss(x, y)
    print("Epoch: ", epoch, "w = ", w, "loss = ", l)
    loss_list.append(l)
    trainnum_list.append(epoch)
print("Predict(after training)", 4, forward(4))  # 训练后x = 4，对应预测的y值

plt.plot(trainnum_list, loss_list)
plt.ylabel("Loss")
plt.xlabel("Epoch")
plt.show()