pytorch系列笔记二:批处理与优化器选择
批处理
批处理对于神经网络的训练是必不可少的,通过对有限数据的shuffle并重新送入模型,因为训练的数据更多了,所以可以提高模型的训练效果
在Pytorch中要使用批处理需要进行如下步骤:
-
利用数据集创建一个
TensorDataset
:-
#### Download and construct CIFAR-10 dataset. train_dataset = torchvision.datasets.CIFAR10(root='../data/', train=True, transform=transforms.ToTensor(), download=False)
-
#### make custom data be a TensorDataset torch_dataset = torch.utils.data.TensorDataset(x, y)
-
# You should build your custom dataset as below. class CustomDataset(torch.utils.data.Dataset): def __init__(self): # TODO # 1. Initialize file paths or list of file names pass def __getitem__(self,index): # TODO # 1.Read one data from file (e.g. using numpy.fromfile,PIL.image.open) # 2. Preprocess the data (e.g. torchvision.Transform). # 3. Return a data pair (e.g. image and label). pass def __len__(self): # You should change 0 to the total size of your dataset return 0 # You can then use the prebuilt data loader custom_dataset = CustomDataset() train_loader = torch.utils.data.DataLoader(dataset=custom_dataset, batch_size=64, shuffle=True)
-
-
使用
TensorDataset
创建DataLoader
:loader = torch.utils.data.DataLoader( dataset=torch_dataset, batch_size=BATCH_SIZE, shuffle=True, # shuffle=False, num_workers=2, )
-
使用
DataLoader
的实例化对象,通过循环一批一批地从数据集中采集样本:for epoch in range(3): for step, (batch_x, batch_y) in enumerate(loader): # TODO:training print('EPOCH:', epoch, '| Step:', step, '| batch x:', batch_x.numpy(), '| batch_y:', batch_y.numpy())
通过下面程序的结果可以理解批处理的作用:
import torch
from torch.utils import data
# BATCH_SIZE = 5
BATCH_SIZE = 8
# 10 samples
x = torch.linspace(1, 10, 10)
y = torch.linspace(10, 1, 10)
torch_dataset = data.TensorDataset(x, y)
loader = data.DataLoader(
dataset=torch_dataset,
batch_size=BATCH_SIZE,
shuffle=True,
# shuffle=False,
num_workers=2,
)
if __name__ == '__main__':
for epoch in range(3):
for step, (batch_x, batch_y) in enumerate(loader):
# TODO:training
print('EPOCH:', epoch, '| Step:', step, '| batch x:',
batch_x.numpy(), '| batch_y:', batch_y.numpy())
输出:
EPOCH: 0 | Step: 0 | batch x: [ 1. 4. 10. 8. 7. 9. 2. 5.] | batch_y: [10. 7. 1. 3. 4. 2. 9. 6.]
EPOCH: 0 | Step: 1 | batch x: [3. 6.] | batch_y: [8. 5.]
EPOCH: 1 | Step: 0 | batch x: [ 2. 9. 8. 3. 5. 4. 10. 6.] | batch_y: [9. 2. 3. 8. 6. 7. 1. 5.]
EPOCH: 1 | Step: 1 | batch x: [7. 1.] | batch_y: [ 4. 10.]
EPOCH: 2 | Step: 0 | batch x: [ 5. 4. 7. 10. 6. 1. 2. 3.] | batch_y: [ 6. 7. 4. 1. 5. 10. 9. 8.]
EPOCH: 2 | Step: 1 | batch x: [8. 9.] | batch_y: [3. 2.]
优化器的选择
越复杂的神经网络,越多的数据会导致我们在训练神经网络所需花费的时间越多,所以需要寻找加速神经网络训练的优化器,使得神经网络的训练变得快起来。
最基础的神经网络训练方法为SGD(随机梯度下降法)
每次使用随机的一个Mini-Batch数据来进行梯度下降,在进行多次Epoch后也可以达到与使用原始梯度下降法一样的效果,而且速度大大提升。
前面有过一次手写实现,可以体现其处理数据的核心逻辑
def dJ_sgd(theta,X_b_i,y_i):
# 只对一个样本做梯度下降处理
return X_b_i.T.dot(X_b_i.dot(theta)-y_i) *2
def sgd