pytorch-backward（）原理详细讲解

张量梯度和雅可比积

在很多情况下，我们有一个标量损失函数，我们需要计算一些参数的梯度。但是，有些情况下输出函数是任意张量。在这种情况下，PyTorch 允许您计算所谓的Jacobian 乘积，而不是实际的梯度。

对于向量函数$\vec{y}=f\left(\vec{x}\right)$,其中$\vec{x}=⟨x_1,...,x_n⟩$ , $\vec{y}=⟨y_1,...,y_m⟩$,利用 Jacobian matrix给出$\vec{y}$关于$\vec{x}$的偏导：
$$J=\left(\begin{array}{ccc}\frac{\partial y_1}{\partial x_1}& \cdots & \frac{\partial y_1}{\partial x_n}\\ ⋮& \ddots & ⋮\\ \frac{\partial y_m}{\partial x_1}& \cdots & \frac{\partial y_m}{\partial x_n}\end{array}\right)$$

PyTorch 允许您对于给定的输入向量$\vec{v}=⟨v_1,...,v_m⟩$计算雅可比乘积 $v\cdot J$，而不是计算雅可比矩阵本身。这是通过使用v作为参数向后调用来实现的。(v的大小应该和原始张量的大小相同，我们要用它来计算乘积)：

$$v\cdot J={\left(\begin{array}{ccc}{v}_1& \cdots & v_m\end{array}\right)}_{1\times m}{\left(\begin{array}{ccc}\frac{\partial y_1}{\partial x_1}& \cdots & \frac{\partial y_1}{\partial x_n}\\ ⋮& \ddots & ⋮\\ \frac{\partial y_m}{\partial x_1}& \cdots & \frac{\partial y_m}{\partial x_n}\end{array}\right)}_{m\times n}={\left(\begin{array}{ccc}\frac{\partial y}{\partial x_1}& \cdots & \frac{\partial y}{\partial x_n}\end{array}\right)}_{1\times n}$$

测试

inp = torch.eye(5, requires_grad=True)
out = (inp+1).pow(2)
out.backward(torch.ones_like(inp), retain_graph=True)
print("First call\n", inp.grad)
out.backward(torch.ones_like(inp), retain_graph=True)
print("\nSecond call\n", inp.grad)
inp.grad.zero_()
out.backward(torch.ones_like(inp), retain_graph=True)
print("\nCall after zeroing gradients\n", inp.grad)

First call
 tensor([[4., 2., 2., 2., 2.],
        [2., 4., 2., 2., 2.],
        [2., 2., 4., 2., 2.],
        [2., 2., 2., 4., 2.],
        [2., 2., 2., 2., 4.]])

Second call
 tensor([[8., 4., 4., 4., 4.],
        [4., 8., 4., 4., 4.],
        [4., 4., 8., 4., 4.],
        [4., 4., 4., 8., 4.],
        [4., 4., 4., 4., 8.]])

Call after zeroing gradients
 tensor([[4., 2., 2., 2., 2.],
        [2., 4., 2., 2., 2.],
        [2., 2., 4., 2., 2.],
        [2., 2., 2., 4., 2.],
        [2., 2., 2., 2., 4.]])

关于非标量损失函数backward（）函数使用例子

计算$y=5x^2$ 每个整数(1，2，…，10)的导数

• 利用for循坏得到x为整数时y的值，并依次作为标量进行导数计算(标量)
• 分别利用v(1,0,…,0)继续进行导数运算(非标量)

x = torch.range(1, 10, 1 ,requires_grad=True)
# x1 = torch.randn(91 ,requires_grad=True)
l = []
for i in x:
    y = 5*i**2
    l.append(y.detach().numpy())

    #compute gradients
    y.backward()

    #print out the gradients.
    print(x.grad)

tensor([10.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])
tensor([10., 20.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])
tensor([10., 20., 30.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])
tensor([10., 20., 30., 40.,  0.,  0.,  0.,  0.,  0.,  0.])
tensor([10., 20., 30., 40., 50.,  0.,  0.,  0.,  0.,  0.])
tensor([10., 20., 30., 40., 50., 60.,  0.,  0.,  0.,  0.])
tensor([10., 20., 30., 40., 50., 60., 70.,  0.,  0.,  0.])
tensor([10., 20., 30., 40., 50., 60., 70., 80.,  0.,  0.])
tensor([10., 20., 30., 40., 50., 60., 70., 80., 90.,  0.])
tensor([ 10.,  20.,  30.,  40.,  50.,  60.,  70.,  80.,  90., 100.])


C:\Users\pandas\AppData\Local\Temp/ipykernel_1728/2678808987.py:1: UserWarning: torch.range is deprecated and will be removed in a future release because its behavior is inconsistent with Python's range builtin. Instead, use torch.arange, which produces values in [start, end).
  x = torch.range(1, 10, 1 ,requires_grad=True)

# x.zero_grad()  
x = torch.range(1, 10, 1 ,requires_grad=True)
y = 5*x.T*x
v = [1]*10
v = torch.tensor(v, dtype=torch.float)
# y.backward(torch.ones_like(y))
y.backward(v)

x.grad

C:\Users\pandas\AppData\Local\Temp/ipykernel_1728/1828972091.py:2: UserWarning: torch.range is deprecated and will be removed in a future release because its behavior is inconsistent with Python's range builtin. Instead, use torch.arange, which produces values in [start, end).
  x = torch.range(1, 10, 1 ,requires_grad=True)





tensor([ 10.,  20.,  30.,  40.,  50.,  60.,  70.,  80.,  90., 100.])

稍复杂的多维度梯度计算

import torch
from torch.autograd import Variable
x=torch.Tensor([[1.,2.,3.],[4.,5.,6.]])
x=Variable(x,requires_grad=True)
y=x+2
z=y*y*3
out=z.mean()

print(x)
print(y)
print(z)
print(out)

tensor([[1., 2., 3.],
        [4., 5., 6.]], requires_grad=True)
tensor([[3., 4., 5.],
        [6., 7., 8.]], grad_fn=<AddBackward0>)
tensor([[ 27.,  48.,  75.],
        [108., 147., 192.]], grad_fn=<MulBackward0>)
tensor(99.5000, grad_fn=<MeanBackward0>)

print(x.grad_fn)
print(y.grad_fn)
print(z.grad_fn)
print(out.grad_fn)

None
<AddBackward0 object at 0x0000013BFE723BE0>
<MulBackward0 object at 0x0000013BFE712FD0>
<MeanBackward0 object at 0x0000013BFE723BE0>

out.backward()
x.grad

tensor([[3., 4., 5.],
        [6., 7., 8.]])

解析：
由梯度计算的链式法则计算out与x。
$$out=\frac{1}{6}\sum _i{z}_i$$
$$z_i=3\times (x_i+2{)}^2$$
$$\frac{\partial out}{\partial x_i}=x_i+2$$
所以
$$\left(\begin{array}{ccc}1& 2& 3\\ 4& 5& 6\end{array}\right)\Rightarrow \left(\begin{array}{ccc}3& 4& 5\\ 6& 7& 8\end{array}\right)$$