Pytorch入门初体验(五)

最新推荐文章于 2024-07-28 09:26:52 发布

爱上北方的冷冬

最新推荐文章于 2024-07-28 09:26:52 发布

阅读量2.5k

点赞数

分类专栏： PyTorch学习笔记

本文链接：https://blog.csdn.net/alawaka2018/article/details/80993197

版权

PyTorch学习笔记专栏收录该内容

4 篇文章 2 订阅

订阅专栏

数据的并行计算（DataParallelism）

在这个教程中，我们将会学习怎样使用多块GPU进行数据并行计算。

在PyTorch中使用GPU是非常简单的，你可以将模型放到一块GPU上：

device = torch.device("cuda:0")
model.to(device)

然后，你可以将所有的tensors复制到GPU上：

mytensor = my_tensor.to(device)

请注意my_tensor.to(device)返回的是my_tensor在GPU上的一个新备份而不是重写my_tensor。你需要重新指定它到一个新的tensor上并且在GPU上使用这个tensor。

PyTorch默认只会使用一块GPU，你可以在多块GPU上跑你的程序通过使你的模型用DataParallel并行计算：

model = nn.DataParallel(model)

下面将会具体的讲解一些细节:

Imports andparameters

导入pytorchmodules并定义参数：

import torch
import torch.nn as nn
from torch.ultis.data import Dataset, Dataloader

#parameter and DataLoaders
input_size = 5
output_size = 2
batch_size = 30
data_size = 100

Device

device = torch.device("cuda:0" if torch.cuda.is_available() else"cpu")

虚拟数据集（Dummy DataSet）

做一个虚拟的或是随机的数据集，你只需要执行getiterm

class RandomDataset(Dataset):
    def __init__(self, size,length):
        self.len = length
        self.data =torch.randn(length, size)      
        def __getitem__(self,index):
            return self.data[index]
     
        def __len__(self):
            return self.len
rand_loader = DataLoader(dataset=RandomDataset(input_size, data_size),
                       batch_size=batch_size, shuffle=True)

简单模型（simple model）

在这个demo中，我们的模型只有一个输入，做一个线性操作，然后有一个输出。但是，你可以在任意的模型上（CNN，RNN，Capsule Net等）使用DataParallel.

class Model(nn.Module):

    # Our model
    def __init__(self, input_size,output_size):
        super(Model,self).__init__()
        self.fc =nn.Linear(input_size, output_size)
    def forward(self, input):
        output = self.fc(input)
        print("\tIn Model:input size", input.size(), "output size", output.size())
        return output

建立一个模型并且数据并行（createmodel and dataparallel）

这是这个教程的核心内容。

首先，你需要建立一个模型实例并且check一下你是否有多块GPU，如果有多块GPU，你就可以使用nn.DataParallel包装模型；然后通过model.to(device)将我们的模型放到GPU上。

model = Model(input_size, output_size)
if torch.cuda.device_count() > 1:
    print("Let's use",torch.cuda.device_count(), "GPUs!")
    model = nn.DataParallel(model)   
model.to(device)

Run the Model

现在我们可以看到输入和输出tensors的sizes:

for data in rand_loader:
    input = data.to(device)
    output =  model(input)
    print("Outside: inputsize", input.size(), "Output_size", output.size())

Out:

In Model: input size torch.Size([30, 5]) output size torch.Size([30, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
        In Model: input size torch.Size([30, 5]) output size torch.Size([30, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
        In Model: input size torch.Size([30, 5]) output size torch.Size([30, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
        In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])

Results

如果你没有GPU或只有一个GPU，当我们batch30 inputs和30 outputs的时候，模型输入和输出都超过30。但是如果我们有多个GPU的话，我们获得的结果如下：

2个GPU：

# on 2 GPUs
Let's use 2 GPUs!
    In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])
    In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
    In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])
    In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
    In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])
    In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
    In Model: input size torch.Size([5, 5]) output size torch.Size([5, 2])
    In Model: input size torch.Size([5, 5]) output size torch.Size([5, 2])
Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])

3个GPUs:

Let's use 3 GPUs!
    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
    In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])

8个GPUs:

Let's use 8 GPUs!
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
    In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])

总结：

DataParallel splits your data automatically and sends job orders tomultiple models on several GPUs. After each model finishes their job,DataParallel collects and merges the results before returning it to you.