Microsoft DeepSpeedExamples 项目教程
1. 项目的目录结构及介绍
Microsoft DeepSpeedExamples 项目的目录结构如下:
DeepSpeedExamples/
├── README.md
├── bing_bert
│ ├── bert_config.json
│ ├── bing_config.json
│ ├── data
│ ├── deepspeed_config.json
│ ├── Dockerfile
│ ├── eval_distill.py
│ ├── inference.py
│ ├── LICENSE
│ ├── pretrain_bert.py
│ ├── requirements.txt
│ ├── run_squad.py
│ └── utils.py
├── cifar
│ ├── data
│ ├── deepspeed_config.json
│ ├── Dockerfile
│ ├── README.md
│ ├── requirements.txt
│ └── train.py
├── imagenet
│ ├── data
│ ├── deepspeed_config.json
│ ├── Dockerfile
│ ├── README.md
│ ├── requirements.txt
│ └── train.py
├── language-modeling
│ ├── data
│ ├── deepspeed_config.json
│ ├── Dockerfile
│ ├── README.md
│ ├── requirements.txt
│ └── train.py
├── reinforcement-learning
│ ├── data
│ ├── deepspeed_config.json
│ ├── Dockerfile
│ ├── README.md
│ ├── requirements.txt
│ └── train.py
└── transformer-xl
├── data
├── deepspeed_config.json
├── Dockerfile
├── README.md
├── requirements.txt
└── train.py
目录结构介绍
bing_bert
: 包含BERT模型的预训练和微调脚本。cifar
: 包含CIFAR-10数据集的训练脚本。imagenet
: 包含ImageNet数据集的训练脚本。language-modeling
: 包含语言模型的训练脚本。reinforcement-learning
: 包含强化学习模型的训练脚本。transformer-xl
: 包含Transformer-XL模型的训练脚本。
每个子目录通常包含以下文件:
deepspeed_config.json
: DeepSpeed配置文件。train.py
: 训练脚本。requirements.txt
: 依赖包列表。Dockerfile
: Docker容器配置文件。
2. 项目的启动文件介绍
每个子目录中的train.py
文件是项目的启动文件。例如,在cifar
目录中,train.py
是启动CIFAR-10训练的脚本。
启动文件示例
以cifar
目录中的train.py
为例:
import deepspeed
import torch
import torchvision
import torchvision.transforms as transforms
# 数据加载
transform = transforms.Compose(
[transforms.ToTensor(),
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
shuffle=True, num_workers=2)
testset = torchvision.datasets.CIFAR10(root='./data', train=False,
download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
shuffle=False, num_workers=2)
classes = ('plane', 'car', 'bird', 'cat',
'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
# 模型定义
net = Net()
# DeepSpeed 配置
ds_config = {
"train_batch_size": 32,
"gradient_accumulation_steps": 1,
"fp16": {
"enabled": True,
"loss_scale": 0,
"loss_scale_window": 1000,
"hysteresis": 2,
"min_loss_scale": 1
}
}
# 初始化 DeepSpeed
model, optimizer, _,