Pytorch笔记(二)

用例子来学习Pytorch

第二篇教程主要是通过例子来介绍Pytorch 教程传送门

文中主要抛出两个关于Pytorch的主要特征
1.An n-dimensional Tensor, similar to numpy but can run on GPUs
一个N纬度的张量，和Numpy很类似但是在GPU上运行
2.Automatic differentiation for building and training neural networks
自动的区分神经网络的构建和训练过程

张量

热身活动：用numpy构建神经网络

# -*- coding: utf-8 -*-
import numpy as np

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = np.random.randn(N, D_in)
y = np.random.randn(N, D_out)

# Randomly initialize weights
w1 = np.random.randn(D_in, H)
w2 = np.random.randn(H, D_out)

learning_rate = 1e-6
for t in range(500):
    # Forward pass: compute predicted y
    h = x.dot(w1)
    h_relu = np.maximum(h, 0)
    y_pred = h_relu.dot(w2)

    # Compute and print loss
    loss = np.square(y_pred - y).sum()
    print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.T.dot(grad_y_pred)
    grad_h_relu = grad_y_pred.dot(w2.T)
    grad_h = grad_h_relu.copy()
    grad_h[h < 0] = 0
    grad_w1 = x.T.dot(grad_h)

    # Update weights
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2

解析：这段代码体现了一个三层神经网络的构建和梯度下降过程，首先用np.rand.randn 构建一个输入和一个真实值，前面64，1000，100，10分别为batch_size，输入的维度大小，隐层的节点数量，输出的纬度大小；
综上我们可以知道输入为 (batch_size,1000) ,隐藏层与输入层之间的参数矩阵w1为 (1000,100) ,隐藏层与输出层之间的参数量为w2为 (100,10)

代码中学习率设置为$10^{-6}$

后面for循环开始就是前向传播和反向传播了，range(500)代表进行500此迭代(Step)，
前向传播的过程就是不断的做矩阵乘法

最终得到的Y_pred的大小为(batch_size,10)

损失函数为误差的平方之和
$$loss=\sum (y_{true}-y_{pred}{)}^2$$
$$los{s}^{\prime }=2(y_{true}-y_{pred})$$

所以在反向传播第一步，输出层的梯度为grad_y_pred = 2.0 * (y_pred - y)
根据链式求导法则,隐层的梯度为grad_w2 = h_relu.T.dot(grad_y_pred)

Torch-Pytorch的基本组件

下面我们是tensor同样实现，前向传播和反向传播(Manually implement)

# -*- coding: utf-8 -*-

import torch


dtype = torch.float
device = torch.device("cpu")
# device = torch.device("cuda:0") # Uncomment this to run on GPU

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)

# Randomly initialize weights
w1 = torch.randn(D_in, H, device=device, dtype=dtype)
w2 = torch.randn(H, D_out, device=device, dtype=dtype)

learning_rate = 1e-6
for t in range(500):
    # Forward pass: compute predicted y
    h = x.mm(w1) #torch.mm就是矩阵乘法
    h_relu = h.clamp(min=0)  #clamp是夹子的意思，这里可以看做，只要小于0的值都会被视作0
    y_pred = h_relu.mm(w2)

    # Compute and print loss
    loss = (y_pred - y).pow(2).sum().item()  #item用于把一个仅含有一个元素的tensor
    												#转化为Python数字,同种还有tolist()
    if t % 100 == 99:
        print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.t().mm(grad_y_pred)
    grad_h_relu = grad_y_pred.mm(w2.t())
    grad_h = grad_h_relu.clone()
    grad_h[h < 0] = 0
    grad_w1 = x.t().mm(grad_h)

    # Update weights using gradient descent
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2

关于clamp函数的示意图

自动求导机制

上面我们已经自己实现了前向传播和反向传播，对于链式求导法则有个初步的应用的，但是对于复杂的网络来说，进行反向传播的算法是困难的(hairy)。

在原版的英文材料里面我们注意到两个概念

1.node (是tensor，我们的数据)
2.edge(是function，我们运算过程中的方法）

backpropagation 反向传播

# -*- coding: utf-8 -*-
import torch

dtype = torch.float
device = torch.device("cpu")
# device = torch.device("cuda:0") # Uncomment this to run on GPU

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random Tensors to hold input and outputs.
# Setting requires_grad=False indicates that we do not need to compute gradients
# with respect to these Tensors during the backward pass.
x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)

# Create random Tensors for weights.
# Setting requires_grad=True indicates that we want to compute gradients with
# respect to these Tensors during the backward pass.
w1 = torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True)
w2 = torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True)

learning_rate = 1e-6
for t in range(500):
    # Forward pass: compute predicted y using operations on Tensors; these
    # are exactly the same operations we used to compute the forward pass using
    # Tensors, but we do not need to keep references to intermediate values since
    # we are not implementing the backward pass by hand.
    y_pred = x.mm(w1).clamp(min=0).mm(w2)

    # Compute and print loss using operations on Tensors.
    # Now loss is a Tensor of shape (1,)
    # loss.item() gets the scalar value held in the loss.
    loss = (y_pred - y).pow(2).sum()
    if t % 100 == 99:
        print(t, loss.item())

    # Use autograd to compute the backward pass. This call will compute the
    # gradient of loss with respect to all Tensors with requires_grad=True.
    # After this call w1.grad and w2.grad will be Tensors holding the gradient
    # of the loss with respect to w1 and w2 respectively.
    loss.backward()

    # Manually update weights using gradient descent. Wrap in torch.no_grad()
    # because weights have requires_grad=True, but we don't need to track this
    # in autograd.
    # An alternative way is to operate on weight.data and weight.grad.data.
    # Recall that tensor.data gives a tensor that shares the storage with
    # tensor, but doesn't track history.
    # You can also use torch.optim.SGD to achieve this.
    with torch.no_grad():
        w1 -= learning_rate * w1.grad
        w2 -= learning_rate * w2.grad

        # Manually zero the gradients after updating weights
        w1.grad.zero_()
        w2.grad.zero_()

有几个点需要注意:
1.计算图(Computation Graph)
就是用来记录一个反向传播过程中的计算图，相当于一个计算流程的记录，可以用于反向传播，文中也提到，我们所有反向传播的梯度被维系在tensor中

2.with toch.no_grad():
有个上面那个概念，下面就很好理解了，更新梯度的过程不是我们反向传播或者正向传播的流程，只是我们为了更新一下系数

3.grad_zero
那么下面就更好解释了，既然参数更细了，之前传出来的梯度就意义不大了，因为是针对之前的系数算出来了，我们要清除一下，用新的，因为Pytorch的梯度在每次计算是累加的

定义新的自动求导函数

# -*- coding: utf-8 -*-
import torch


class MyReLU(torch.autograd.Function):
    """
    We can implement our own custom autograd Functions by subclassing
    torch.autograd.Function and implementing the forward and backward passes
    which operate on Tensors.
    """

    @staticmethod
    def forward(ctx, input):
        """
        In the forward pass we receive a Tensor containing the input and return
        a Tensor containing the output. ctx is a context object that can be used
        to stash information for backward computation. You can cache arbitrary
        objects for use in the backward pass using the ctx.save_for_backward method.
        """
        ctx.save_for_backward(input)
        return input.clamp(min=0)

    @staticmethod
    def backward(ctx, grad_output):
        """
        In the backward pass we receive a Tensor containing the gradient of the loss
        with respect to the output, and we need to compute the gradient of the loss
        with respect to the input.
        """
        input, = ctx.saved_tensors
        grad_input = grad_output.clone()
        grad_input[input < 0] = 0
        return grad_input


dtype = torch.float
device = torch.device("cpu")
# device = torch.device("cuda:0") # Uncomment this to run on GPU

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random Tensors to hold input and outputs.
x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)

# Create random Tensors for weights.
w1 = torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True)
w2 = torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True)

learning_rate = 1e-6
for t in range(500):
    # To apply our Function, we use Function.apply method. We alias this as 'relu'.
    relu = MyReLU.apply

    # Forward pass: compute predicted y using operations; we compute
    # ReLU using our custom autograd operation.
    y_pred = relu(x.mm(w1)).mm(w2)

    # Compute and print loss
    loss = (y_pred - y).pow(2).sum()
    if t % 100 == 99:
        print(t, loss.item())

    # Use autograd to compute the backward pass.
    loss.backward()

    # Update weights using gradient descent
    with torch.no_grad():
        w1 -= learning_rate * w1.grad
        w2 -= learning_rate * w2.grad

        # Manually zero the gradients after updating weights
        w1.grad.zero_()
        w2.grad.zero_()

关于相关函数的解释请参见

Pytorch笔记04-自定义torch.autograd.Function

非常感谢大神的讲解，这里再一次验证了，链式求导法则在反向传播中发挥得玄妙

ctx可以代替为self，说明在python中指向自身对象的参数不一定限制死名字是self，可以自定义