02-梯度下降、随机梯度下降

Def

• 损失函数loss：对单个样本
• 代价函数cost：对整个训练集 MSE均方误差
• 梯度（Batch gradient）：cost对权重的导数
• 随机梯度(Stochastic gradient)：loss对权重的导数
• 凸函数：任取两个点，画一条直线，直线都在曲线上方，则为凸函数。凸函数有全局最优。
• BGD：收敛到局部最优
• SGD：更容易收敛到全局最优，但时间复杂度大
• GD 和 SGD 的折中：mini-batchGD （深度学习中常用）
• 鞍点：导数那里等于零（某段曲线斜率为0，平了）

代码分析

02-BGD

# -*- coding:utf-8 -*-
# Abstract : 模型：y^ = wx BGD

import matplotlib.pyplot as plt

x_data = [1.0, 2.0, 3.0]
y_data = [2.0, 4.0, 6.0]

w = 1.0

def forward(x):
    return w * x

def cost(xs, ys):
    cost = 0.0
    for x,y in zip(xs, ys):
        y_pred = forward(x)
        loss = (y_pred - y) ** 2
        cost += loss
    return cost/len(xs)

def gradient(xs,ys):
    grad = 0.0
    for x,y in zip(xs,ys):
        y_pred = forward(x)
        grad += 2 * x * (y_pred - y)
    return grad/len(xs)

def draw():
    plt.plot(ep_lst,cost_lst)
    plt.xlabel('epoch')
    plt.ylabel('cost')
    plt.show()

cost_lst = []
ep_lst = []
learning_rate = 0.01
for epoch in range(0,101,1):
    ep_lst.append(epoch)
    cost_val = cost(x_data, y_data)
    cost_lst.append(cost_val)
    grad_val = gradient(x_data, y_data)
    w = w - learning_rate * grad_val
    print('epoch:', epoch, ' w:', '%.4f'%w, ' cost:','%.4f'%cost_val)
draw()

结果：

02-SGD

# -*- coding:utf-8 -*-
# Abstract : 模型：y^ = wx SGD

import matplotlib.pyplot as plt

x_data = [1.0, 2.0, 3.0]
y_data = [2.0, 4.0, 6.0]

w = 1.0

def forward(x):
    return w * x

def loss(x, y):
    y_pred = forward(x)
    loss = (y_pred - y) ** 2
    return loss

def gradient(x,y):
    y_pred = forward(x)
    s_grad = 2 * x * (y_pred - y)
    return s_grad

def draw():
    plt.plot(ep_lst,loss_lst)
    plt.xlabel('epoch')
    plt.ylabel('cost')
    plt.show()

loss_lst = []
ep_lst = []
learning_rate = 0.01
for epoch in range(0,101,1):
    for x,y in zip(x_data, y_data):
        ep_lst.append(epoch)
        s_grad = gradient(x,y)
        w = w - learning_rate * s_grad
        loss_val = loss(x,y)
        loss_lst.append(loss_val)
    print('epoch:', epoch, ' w:', '%.4f'%w, ' loss:','%.4f'%loss_val)
draw()

结果：