Pytorch中x.data()与x.detach()的区别

Pytorch中x.data()与x.detach()的区别

作用

• 阻断梯度回传

class TestDetach(nn.Module):
    def __init__(self, InDim, HiddenDim, OutDim):
        super().__init__()
        self.layer1 = nn.Linear(InDim, HiddenDim, False)
        self.layer2 = nn.Linear(HiddenDim, OutDim, False)

    def forward(self, x, DetachLayer1):
        x = torch.relu(self.layer1(x))
        x = x.detach()
        # x = x.data()
        x = self.layer2(x)
        return x

• 两层线性层，第一层的输出后detach,那么第一层的参数永远不会更新

相同点

• x.data()或x.detach()均会返回与x相同数据的Tensor，并且这个Tensor与原来的Tensor共用内存，一者改变，另一者也会跟着改变，并且新的tensor的requires_grad = False

不同点

• 当c = x.data()或c = x.detach()中的c未被改变时，x.data()与x.detach()并没有什么区别，但是当c发生改变时，x.data()并不会报错，因而导致此时即使求导结果错误，也很难发现，相比之下，x.detach()则会报错，显示需要计算的张量已经被更改

代码示例

import torch
 
a = torch.tensor([1, 2, 3.], requires_grad=True)
print(a)
out = a.tanh()
print(out)
c = out.data  # 需要走注意的是，通过.data “分离”得到的的变量会和原来的变量共用同样的数据，而且新分离得到的张量是不可求导的，c发生了变化，原来的张量也会发生变化
c.zero_() # 改变c的值，原来的out也会改变
print(c.requires_grad)
print(c)
print(out.requires_grad)
print(out)
print("----------------------------------------------")
 
out.sum().backward()  # 对原来的out求导，
print(a.grad)  # 不会报错，但是结果却并不正确
 
#输出
tensor([1., 2., 3.], requires_grad=True)
tensor([0.7616, 0.9640, 0.9951], grad_fn=<TanhBackward>)
False
tensor([0., 0., 0.])
True
tensor([0., 0., 0.], grad_fn=<TanhBackward>)
----------------------------------------------
tensor([1., 1., 1.])

import torch
 
a = torch.tensor([1, 2, 3.], requires_grad=True)
print(a)
out = a.tanh()
print(out)
c = out.detach()  # 需要走注意的是，通过.detach() “分离”得到的的变量会和原来的变量共用同样的数据，而且新分离得到的张量是不可求导的，c发生了变化，原来的张量也会发生变化
c.zero_()  # 改变c的值，原来的out也会改变
print(c.requires_grad)
print(c)
print(out.requires_grad)
print(out)
print("----------------------------------------------")
 
out.sum().backward()  # 对原来的out求导，
print(a.grad)  # 此时会报错，错误结果参考下面,显示梯度计算所需要的张量已经被“原位操作inplace”所更改了。
 
# 输出
tensor([1., 2., 3.], requires_grad=True)
tensor([0.7616, 0.9640, 0.9951], grad_fn=<TanhBackward>)
False
tensor([0., 0., 0.])
True
tensor([0., 0., 0.], grad_fn=<TanhBackward>)
----------------------------------------------
Traceback (most recent call last):
  File "E:/python/TCL/entropy_coding_project/test_code/test27.py", line 15, in <module>
    out.sum().backward()  # 对原来的out求导，
  File "D:\ProgramData\Anaconda3\lib\site-packages\torch\tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "D:\ProgramData\Anaconda3\lib\site-packages\torch\autograd\__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3]], which is output 0 of TanhBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

拓展

• 关于pytorch中使用detach并不能阻止参数更新这档子事儿
• pytorch中detach()和detach_()的用法和区别