Pytorch学习笔记（一）

1、
Tensor

• 对于tensor,如果requires_grad=True，整个计算会被跟踪，等反传结束后，梯度值会保存在.grad中
• Tensor和Function内部交联，构成一个图
• 对于Variable，它由某个操作完成，那么.grad_fn将会指向完成这个变量的操作

x = torch.ones(2, 2, requires_grad=True)
y = x + 2
y.backward(torch.ones(2, 2), retain_graph=True)
# the retain_variables flag will prevent the internal buffers from being freed
print(x.grad)       #tensor([[ 1.,  1.],
                    #[ 1.,  1.]])
print(y.grad)     #None  #y是中间变量，所以打印他的grad输出将会是None
print(y.requires_grad) #True 对于Tensor都是有requires_grad性质的

2
Tensor和numpy数组几乎是一致的，他对于深度学习，计算图，梯度一无所知，只是一个数组存储数据。Tensor和numpy的区别在于Tensor可以应用在GPU上和CPU上，而numpy只能在CPU上面。

3、pytorch自动微分机制，只需要定义前项动态图，每一个节点都是一个Tensor,如果一个Tensor的requires_grad=True，那么这个Tensor的grad中将会存储相应的梯度

4、
因为图的建立是动态的，所以可以使用python的control-flow操作子例如循环和if语句定义整个模型的前向传递，一个模块可以被使用多次，参数共享

import random
import torch


class DynamicNet(torch.nn.Module):
    def __init__(self, D_in, H, D_out):
        """
        In the constructor we construct three nn.Linear instances that we will use
        in the forward pass.
        """
        super(DynamicNet, self).__init__()
        self.input_linear = torch.nn.Linear(D_in, H)
        self.middle_linear = torch.nn.Linear(H, H)
        self.output_linear = torch.nn.Linear(H, D_out)

    def forward(self, x):
        """
        For the forward pass of the model, we randomly choose either 0, 1, 2, or 3
        and reuse the middle_linear Module that many times to compute hidden layer
        representations.

        Since each forward pass builds a dynamic computation graph, we can use normal
        Python control-flow operators like loops or conditional statements when
        defining the forward pass of the model.

        Here we also see that it is perfectly safe to reuse the same Module many
        times when defining a computational graph. This is a big improvement from Lua
        Torch, where each Module could be used only once.
        """
        h_relu = self.input_linear(x).clamp(min=0)
        for _ in range(random.randint(0, 3)):
            h_relu = self.middle_linear(h_relu).clamp(min=0)
        y_pred = self.output_linear(h_relu)
        return y_pred

TensorFlow中参数的更新是在整个计算图之中的，但是pytorch的参数更新是在计算图之外进行的

5、
Varibale && Tensor
Variable封装了tensor，支持在tensor上面的操作，可以通过backward求梯度，它由三个属性，data,grad,grad_fn(旧版本是creator)

6、
torch.from_numpy将numpy转化为Tensor

7
对于不是nn.Module构造的模型的话，模型输入的变量可以不设置requires_grad=True,但是对于其他的情形必须要设置输入tensor是requires_grad(个人理解而已)，下面的代码将会报错，因为输入的tensor不是requires_grad=True，直接导致y和z都不是requires_grad=True

import torch
#x = torch.ones(2, 2, requires_grad=True)
from torch.autograd import Variable
#x = Variable(torch.ones(2,2),requires_grad=True)
x = torch.ones(2, 2)
y = x + 2

z = y * y * 3
out = z.mean()
out.backward()

8、torch pytorch的函数区别
Pytorch is very similar to nngraph in LuaTorch, except that you dont have Cadd, Cmul or any of the table layers. Its the normal +, * operator.