ModelArts_自定义镜像_Pytorch1.5训练LeNet

PyTorch1.5训练LeNet

文档描述

本样例使用成都人工智能计算中心（以下简称智算中心）ModelArts平台，通过PyTorch自定义镜像训练LeNet，为大家学习ModelArts提供参考。

前置条件

请检查以下条件要求是否满足，如不满足请按照备注进行相应处理。

条件	要求	备注
PyTorch镜像版本	>=21.0.3	PyTorch镜像为Ascend Hub镜像，请参考容器镜像服务指南下载并上传镜像至智算中心镜像服务
OBS Browser+	安装	请参考对象存储服务指南完成下载安装并登录，方便后续上传代码、数据集等
模型迁移	迁移完成	请参考PyTorch网络模型移植&训练指南完成NPU版PyTorch的网络模型迁移，可在modelarts调试

训练环境

• 训练平台：成都人工智能计算中心-ModelArts服务
• 底层硬件资源：Atlas 900 集群
• 训练镜像：ascend-pytorch-arm-modelarts
• 训练框架：PyTorch1.5
• CANN版本：5.0.3

LeNet描述

LeNet是1998年提出的一种典型的卷积神经网络。它被用于数字识别并取得了巨大的成功。

论文： Y.Lecun, L.Bottou, Y.Bengio, P.Haffner.Gradient-Based Learning Applied to Document Recognition.Proceedings of the IEEE.1998.

模型架构

LeNet非常简单，包含5层，由2个卷积层和3个全连接层组成。

数据集

使用的数据集：FashionMNIST（使用torchvision.datasets下载）

• 数据集大小：52.4M，共10个类，6万张 28*28图像
  • 训练集：6万张图像
  • 测试集：1万张图像
• 数据格式：二进制文件
• 目录结构如下：

└─data
    ├──FashionMNIST
        ├──raw
            ├──t10k-images-idx3-ubyte
            ├──t10k-labels.idx1-ubyte
            ├──train-images.idx3-ubyte
            ├──train-labels.idx1-ubyte

数据拷贝

自定义镜像训练过程中，OBS桶中的训练代码会自动上传至ModelArts分配的环境中，但数据集需手动拷贝（当前版本训练日志和保存的模型文件也需手动拷贝至OBS桶中），具体文档参考MoXing相关API。

# modelarts dataset path
data_path = '/cache/data'
# copy dataset to modelarts from ob，useage：mox.file.copy_parallel('obs://[bucket_name]/[data_dir]', [modelarts_data_dir])
mox.file.copy_parallel('obs://wesley/models/lenet_torch1.5/data', data_path)
# model save path
weight_path = '/cache/weights'
# mkdir on modelarts
os.mkdir(weight_path)


# copy train log to obs from modelarts
mox.file.copy('/tmp/log/train.log',
              'obs://wesley/models/lenet_torch1.5/train.log')
# copy models to obs from modelarts
mox.file.copy_parallel(weight_path, 'obs://wesley/models/lenet_torch1.5/weights')

注：ModelArts会挂载硬盘至“/cache”目录，Ascend规格资源该目录大小为3T，用户可以使用此目录来储存临时文件，如数据集和保存的模型文件等。

完整代码

代码基于PyTorch官网教程中的Quickstart修改，代码中的路径请自行修改

import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision import datasets
from torchvision.transforms import ToTensor
import os
import moxing as mox

# Define model
class LeNet5(nn.Module):
    def __init__(self):
        super().__init__()
        # input_shape 28*28
        self.conv1 = nn.Sequential(
            nn.Conv2d(in_channels=1, out_channels=6,
                      kernel_size=5, stride=1),
            nn.MaxPool2d(kernel_size=2)
        )
        self.conv2 = nn.Sequential(
            nn.Conv2d(in_channels=6, out_channels=16,
                      kernel_size=5, stride=1),
            nn.MaxPool2d(kernel_size=2)
        )
        self.fc1 = nn.Sequential(
            nn.Linear(in_features=4 * 4 * 16, out_features=120)
        )
        self.fc2 = nn.Sequential(
            nn.Linear(in_features=120, out_features=84)
        )
        self.fc3 = nn.Sequential(
            nn.Linear(in_features=84, out_features=10)
        )

    def forward(self, input):
        conv1_output = self.conv1(input)  # [28,28,1]
        conv2_output = self.conv2(conv1_output)
        conv2_output = conv2_output.view(-1, 4 * 4 * 16)
        fc1_output = self.fc1(conv2_output)
        fc2_output = self.fc2(fc1_output)
        output = self.fc3(fc2_output)
        return output



data_path = '/cache/data'
weight_path = '/cache/weights'
batch_size = 64
epochs = 5

# if dataset in obs, use mox.file.copy() to copy dataset to modelarts
mox.file.copy_parallel('obs://wesley/models/lenet_torch1.5/data', data_path)
os.mkdir(weight_path)

# Download training data from open datasets.
training_data = datasets.FashionMNIST(
    root=data_path,
    train=True,
    download=False,
    transform=ToTensor(),
)

# Download test data from open datasets.
test_data = datasets.FashionMNIST(
    root=data_path,
    train=False,
    download=False,
    transform=ToTensor(),
)

# Create data loaders.
train_dataloader = DataLoader(training_data, batch_size=batch_size)
test_dataloader = DataLoader(test_data, batch_size=batch_size)

for X, y in test_dataloader:
    print(f"Shape of X [N, C, H, W]: {X.shape}")
    print(f"Shape of y: {y.shape} {y.dtype}")
    break

    
# using ascend device 1p
device = "cpu" if not torch.npu.is_available() else f"npu:{torch.npu.current_device()}"
torch.npu.set_device(device)
print(f"Using {device} device")

model = LeNet5().to(device)
print(model)

# loss function
loss_fn = nn.CrossEntropyLoss()

# optimizer
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

# define train function

def train(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    model.train()
    for batch, (X, y) in enumerate(dataloader):
        X, y = X.to(device), y.to(device)

        # Compute prediction error
        pred = model(X)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")

# define test function

def test(dataloader, model, loss_fn):
    size = len(dataloader.dataset)
    num_batches = len(dataloader)
    model.eval()
    test_loss, correct = 0, 0
    with torch.no_grad():
        for X, y in dataloader:
            X, y = X.to(device), y.to(device)
            pred = model(X)
            test_loss += loss_fn(pred, y).item()
            correct += (pred.argmax(1) == y).type(torch.float).sum().item()
    test_loss /= num_batches
    correct /= size
    print(
        f"Test Error: \n Accuracy: {(100 * correct):>0.1f}%, Avg loss: {test_loss:>8f} \n")

# training

try:
    for t in range(epochs):
        print(f"Epoch {t + 1}\n-------------------------------")
        train(train_dataloader, model, loss_fn, optimizer)
        torch.save(model.state_dict(), f"{weight_path}/lenet_{t + 1}.pth")
        print(f"saved model lenet_{t + 1}.pth success")
        # copy models to obs from modelarts
        mox.file.copy_parallel(weight_path, 'obs://wesley/models/lenet_torch1.5/weights')
        test(test_dataloader, model, loss_fn)
    print("Done!")

finally:
    # copy train log to obs from modelarts
    mox.file.copy('/tmp/log/train.log', 'obs://wesley/models/lenet_torch1.5/train.log')

训练过程

创建训练作业

当前新版训练作业无法使用自定义镜像（非华为云ModelArts），请使用旧版训练作业，步骤如下：

1.算法来源：

选择自定义

2.镜像地址：

先择前置条件中的PyTorch21.0.3镜像

3.代码目录：

选择OBS桶中的训练代码目录

4.运行命令：

默认启动命令如下，可在本地运行PyTorch21.0.3镜像，在/home/work/run_train.sh中查看或修改。

/bin/bash /home/work/run_train.sh ${obs-code-path} ${the-base-name-of-obs-code-path}/${boot-file} '/tmp/log/train.log' ${python_file_parameter}

前两个参数不可修改，前五个参数的顺序不可修改，示例：

/bin/bash /home/work/run_train.sh 'obs://training-bucket/ascend-tf-1.15/resnet50/' 'resnet50/train.py' '/tmp/log/train.log' --'data_url'='obs://training-bucket/cifar-10/'

5.数据来源

数据集通过MoXing手动拷贝，此处可选择OBS桶中训练代码目录下的空文件夹

6.训练输出位置

训练日志和模型文件通过MoXing手动拷贝，此处可选择OBS桶中训练代码目录下的空文件夹

7.环境变量

训练中如需添加环境变量，点击增加即可

8.作业日志路径

该日志为训练过程中Ascend产生的plog日志，选择OBS桶中指定的文件夹即可

9.资源池

按照所需资源选择后点击创建作业即可

其他Python依赖

如果环境中没有训练代码所需Python依赖，在OBS桶中训练代码目录下上传requirements.txt，在其中添加所需依赖及版本即可

查看训练结果

查看训练日志

...
...
Epoch 5
-------------------------------
loss: 1.047490 [ 0/60000]
loss: 1.055037 [ 6400/60000]
loss: 0.821756 [12800/60000]
loss: 0.984526 [19200/60000]
loss: 0.920349 [25600/60000]
loss: 0.918822 [32000/60000]
loss: 0.933400 [38400/60000]
loss: 0.944582 [44800/60000]
loss: 0.886811 [51200/60000]
loss: 0.894010 [57600/60000]
saved model lenet_5.pth success
Test Error:
Accuracy: 67.5%, Avg loss: 0.883350
Done!
THPModule_npu_shutdown success.
[Modelarts Service Log]2022-05-16 09:51:58,597 - INFO - Begin destroy training processes
[Modelarts Service Log]2022-05-16 09:51:58,598 - INFO - proc-rank-0-device-0 (pid: 121) has exited
[Modelarts Service Log]2022-05-16 09:51:58,598 - INFO - End destroy training processes
...
...
[ModelArts Service Log][INFO][2022/05/16 09:51:59]: env MA_OUTPUTS is not found, skip the outputs handler
[ModelArts Service Log]modelarts-pipe: total length: 184
[Modelarts Service Log]Training end at 2022-05-16-09:52:00
[Modelarts Service Log]Training completed.

查看拷贝文件

训练结束后，查看OBS训练代码目录下模型、日志等是否拷贝成功