Pytorch入门初体验(五)

最新推荐文章于 2024-04-21 16:55:05 发布

爱上北方的冷冬

最新推荐文章于 2024-04-21 16:55:05 发布

阅读量2.4k

点赞数

分类专栏： PyTorch学习笔记

本文链接：https://blog.csdn.net/Alawaka2018/article/details/80993197

版权

PyTorch学习笔记专栏收录该内容

4 篇文章 2 订阅

订阅专栏

数据的并行计算（DataParallelism）

在这个教程中，我们将会学习怎样使用多块GPU进行数据并行计算。

在PyTorch中使用GPU是非常简单的，你可以将模型放到一块GPU上：

device = torch.device("cuda:0")
model.to(device)

然后，你可以将所有的tensors复制到GPU上：

mytensor = my_tensor.to(device)

请注意my_tensor.to(device)返回的是my_tensor在GPU上的一个新备份而不是重写my_tensor。你需要重新指定它到一个新的tensor上并且在GPU上使用这个tensor。

PyTorch默认只会使用一块GPU，你可以在多块GPU上跑你的程序通过使你的模型用DataParallel并行计算：

model = nn.DataParallel(model)

下面将会具体的讲解一些细节:

Imports andparameters

导入pytorchmodules并定义参数：

import torch
import torch.nn as nn
from torch.ultis.data import Dataset, Dataloader

#parameter and DataLoaders
input_size = 5
output_size = 2
batch_size = 30
data_size = 100

Device

device = torch.device("cuda:0" if torch.cuda.is_available() else"cpu")

虚拟数据集（Dummy DataSet）

做一个虚拟的或是随机的数据集，你只需要执行getiterm

class RandomDataset(Dataset):
    def __init__(self, size,length):
        self.len = length
        self.data =torch.randn(length, size)      
        def __getitem__(self,index):
            return self.data[index]
     
        def __len__(self):
            return self.len
rand_loader = DataLoader(dataset=RandomDataset(input_size, data_size),
                       batch_size=batch_size, shuffle=True)

简单模型（simple model）

在这个demo中，我们的模型只有一个输入，做一个线性操作，然后有一个输出。但是，你可以在任意的模型上（CNN，RNN，Capsule Net等）使用DataParallel.

class Model(nn.Module):

    # Our model
    def __init__(self, input_size,output_size):
        super(Model,self).__init__()
        self.fc =nn.Linear(input_size, output_size)
    def forward(self, input):
        output = self.fc(input)
        print("\tIn Model:input size", input.size(), "output size", output.size())
        return output

建立一个模型并且数据并行（createmodel and dataparallel）

这是这个教程的核心内容。

首先，你需要建立一个模型实例并且check一下你是否有多块GPU，如果有多块GPU，你就可以使用nn.DataParallel包装模型；然后通过model.to(device)将我们的模型放到GPU上。

model = Model(input_size, output_size)
if torch.cuda.device_count() > 1:
    print("Let's use",torch.cuda.device_count(), "GPUs!")
    model = nn.DataParallel(model)   
model.to(device)

Run the Model

现在我们可以看到输入和输出tensors的sizes:

for data in rand_loader:
    input = data.to(device)
    output =  model(input)
    print("Outside: inputsize", input.size(), "Output_size", output.size())

Out:

In Model: input size torch.Size([30, 5]) output size torch.Size([30, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
        In Model: input size torch.Size([30, 5]) output size torch.Size([30, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
        In Model: input size torch.Size([30, 5]) output size torch.Size([30, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
        In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])

Results

如果你没有GPU或只有一个GPU，当我们batch30 inputs和30 outputs的时候，模型输入和输出都超过30。但是如果我们有多个GPU的话，我们获得的结果如下：

2个GPU：

# on 2 GPUs
Let's use 2 GPUs!
    In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])
    In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
    In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])
    In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
    In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])
    In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
    In Model: input size torch.Size([5, 5]) output size torch.Size([5, 2])
    In Model: input size torch.Size([5, 5]) output size torch.Size([5, 2])
Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])

3个GPUs:

Let's use 3 GPUs!
    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])

8个GPUs:

Let's use 8 GPUs!
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])

总结：

DataParallel splits your data automatically and sends job orders tomultiple models on several GPUs. After each model finishes their job,DataParallel collects and merges the results before returning it to you.

爱上北方的冷冬

关注

0
点赞
踩
4

收藏

觉得还不错? 一键收藏
0
评论
Pytorch入门初体验(五)

数据的并行计算（DataParallelism）在这个教程中，我们将会学习怎样使用多块GPU进行数据并行计算。在PyTorch中使用GPU是非常简单的，你可以将模型放到一块GPU上：device = torch.device("cuda:0")model.to(device)然后，你可以将所有的tensors复制到GPU上：mytensor = my_tensor.to(device)请注意my_...
复制链接

扫一扫