对于线性回归,损失函数有:
L
(
w
,
b
)
=
1
n
∑
i
=
1
n
l
(
i
)
L(\mathbf{w}, b) =\frac{1}{n}\sum_{i=1}^n l^{(i)}
L(w,b)=n1i=1∑nl(i)
(
w
,
b
)
=
1
n
∑
i
=
1
n
1
2
(
w
⊤
x
(
i
)
+
b
−
y
(
i
)
)
2
(\mathbf{w}, b) =\frac{1}{n} \sum_{i=1}^n \frac{1}{2}\left(\mathbf{w}^\top \mathbf{x}^{(i)} + b - y^{(i)}\right)^2
(w,b)=n1i=1∑n21(w⊤x(i)+b−y(i))2
优化的方向是:
(
w
,
b
)
←
(
w
,
b
)
−
η
∣
B
∣
∑
i
∈
B
∂
(
w
,
b
)
l
(
i
)
(
w
,
b
)
(w,b)←(w,b)−η|B|∑i∈B∂(w,b)l(i)(w,b)
(w,b)←(w,b)−η∣B∣∑i∈B∂(w,b)l(i)(w,b)
学习率:
η
η
η 代表在每次优化中,能够学习的步长的大小
批量大小:
B
B
B 是小批量计算中的批量大小batch size
优化函数的有以下两个步骤:
(i)初始化模型参数,一般来说使用随机初始化;
(ii)我们在数据上迭代多次,通过在负梯度方向移动参数来更新每个参数。
import torch
from torch import nn
import torch.utils.data as Data
from IPython import display
from matplotlib import pyplot as plt
import numpy as np
import random
# set input feature number
num_inputs = 2
# set example number
num_examples = 1000
# set true weight and bias in order to generate corresponded label
true_w = [2, -3.4]
true_b = 4.2
#randn函数返回一个张量,包含了从标准正态分布中抽取的一组随机数。
features = torch.tensor(np.random.normal(0, 1, (num_examples, num_inputs)), dtype=torch.float)
#用随机数据通过线性运算得到数据集
labels = true_w[0] * features[:, 0] + true_w[1] * features[:, 1] + true_b
#为这个数据集添加噪声
labels += torch.tensor(np.random.normal(0, 0.01, size=labels.size()), dtype=torch.float)
#定义数据读入函数
import torch.utils.data as Data
batch_size = 10
# combine featues and labels of dataset
dataset = Data.TensorDataset(features, labels)
# put dataset into DataLoader
data_iter = Data.DataLoader(
dataset=dataset, # torch TensorDataset format
batch_size=batch_size, # mini batch size
shuffle=True, # whether shuffle the data or not
num_workers=2, # read data in multithreading
)
#定义线性模型
class LinearNet(nn.Module):
def __init__(self, n_feature):
super(LinearNet, self).__init__() # call father function to init
self.linear = nn.Linear(n_feature, 1) # function prototype: `torch.nn.Linear(in_features, out_features, bias=True)`
def forward(self, x):
y = self.linear(x)
return y
net = LinearNet(num_inputs)
print(net)
#定义损失函数
loss = nn.MSELoss()
#定义优化函数。采用的是随机梯度下降SGD
import torch.optim as optim
optimizer = optim.SGD(net.parameters(), lr=0.03) # built-in random gradient descent function
print(optimizer) # function prototype: `torch.optim.SGD(params, lr=, momentum=0, dampening=0, weight_decay=0, nesterov=False)`
#训练
num_epochs = 3
for epoch in range(1, num_epochs + 1):
for X, y in data_iter:
output = net(X)
l = loss(output, y.view(-1, 1))
optimizer.zero_grad() # reset gradient, equal to net.zero_grad()
l.backward()
optimizer.step()
print('epoch %d, loss: %f' % (epoch, l.item()))
需要关注的是,pytorch创建网络模型的方法有很多种,下面是三种快速搭建模型的方法
# method one
net = nn.Sequential(
nn.Linear(num_inputs, 1)
# other layers can be added here
)
# method two
net = nn.Sequential()
net.add_module('linear', nn.Linear(num_inputs, 1))
# net.add_module ......
# method three
from collections import OrderedDict
net = nn.Sequential(OrderedDict([
('linear', nn.Linear(num_inputs, 1))
# ......
]))