详解PyTorch中的copy_()函数、detach()函数、detach_()函数和clone()函数

参考链接: copy_(src, non_blocking=False) → Tensor
参考链接: detach()
参考链接: detach_()
参考链接: clone() → Tensor

总结:

使用clone():
解释说明: 返回一个原张量的副本,同时不破换计算图,它能够维持反向传播计算梯度,
并且两个张量不共享内存.一个张量上值的改变不影响另一个张量.



使用copy_():
解释说明: 比如x4.copy_(x2), 将x2的数据复制到x4,并且会
修改计算图,使得反向传播自动计算梯度时,计算出x4的梯度后
再继续前向计算x2的梯度. 注意,复制完成之后,两者的值的改变互不影响,
因为他们并不共享内存.



使用detach():
解释说明: 比如x4 = x2.detach(),返回一个和原张量x2共享内存的新张量x4,
两者的改动可以相互可见, 一个张量上的值的改动会影响到另一个张量.
返回的新张量和计算图相互独立,即新张量和计算图不再关联,
因此也无法进行反向传播计算梯度.即从计算图上把这个张量x2拆
卸detach下来,非常形象.




使用detach_():
解释说明: detach()的原地操作版本,功能和detach()类似.
比如x4 = x2.detach_(),其实x2和x4是同一个对象,返回的是self,
x2和x4具有相同的id()值.

以下是手动绘制的计算图(其中的x3部分暂时不用,请读者暂时忽略和x3相关的部分):
在这里插入图片描述

1.最简单的情况:x4 = x2
解释说明: 无需做过多解释

import torch
torch.manual_seed(seed=20200910)
x0 = torch.randn(3, 5, requires_grad=True)
bias = torch.randn(3, 5, requires_grad=True)
x3 = torch.randn(3, 5, requires_grad=False)
x4 = torch.randn(3, 5, requires_grad=False)
x1 = 3 * x0 + 8
x2 = 4 * x1 ** 2 + 5 * x1 + 7

#
# x3 = x2.clone()
# x3 = x3.copy_(x2)
# x3 = x2.detach()
# x3 = x2.detach_()
# x3 = x2

#
# x4 = x2.clone()
# x4 = x4.copy_(x2)
# x4 = x2.detach()
# x4 = x2.detach_()
x4 = x2

#
x5 = 9 * x4 ** 3 + 7 * x4 ** 2 + 3 * x4 + 5 + bias
loss = torch.mean(x5)
print("loss =",loss)
loss.backward()
print("输出梯度信息:")
print(x0.grad)

控制台输出结果:

Windows PowerShell
版权所有 (C) Microsoft Corporation。保留所有权利。

尝试新的跨平台 PowerShell https://aka.ms/pscore6

加载个人及系统配置文件用了 899 毫秒。
(base) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> conda activate ssd4pytorch1_2_0
(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq>  & 'D:\Anaconda3\envs\ssd4pytorch1_2_0\python.exe' 'c:\Users\chenxuqi\.vscode\extensions\ms-python.python-2020.12.424452561\pythonFiles\lib\python\debugpy\launcher' '49618' '--' 'c:\Users\chenxuqi\Desktop\News4cxq\test4cxq\test25.py'
loss = tensor(9.6161e+08, grad_fn=<MeanBackward0>)
输出梯度信息:
tensor([[5.4391e+07, 1.7363e+07, 1.3316e+08, 3.6766e+05, 2.4903e+07],
        [5.1519e+08, 1.5781e+08, 3.6020e+05, 1.7468e+08, 1.2224e+08],
        [4.6100e+07, 9.8575e+07, 6.9595e+05, 2.4009e+07, 1.6252e+07]])
(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq>

2.使用clone()的情况:x4 = x2.clone()
解释说明: 返回一个原张量的副本,同时不破换计算图,它能够维持反向传播计算梯度.

import torch
torch.manual_seed(seed=20200910)
x0 = torch.randn(3, 5, requires_grad=True)
bias = torch.randn(3, 5, requires_grad=True)
x3 = torch.randn(3, 5, requires_grad=False)
x4 = torch.randn(3, 5, requires_grad=False)
x1 = 3 * x0 + 8
x2 = 4 * x1 ** 2 + 5 * x1 + 7

#
# x3 = x2.clone()
# x3 = x3.copy_(x2)
# x3 = x2.detach()
# x3 = x2.detach_()
# x3 = x2

#
x4 = x2.clone()
# x4 = x4.copy_(x2)
# x4 = x2.detach()
# x4 = x2.detach_()
# x4 = x2

#
x5 = 9 * x4 ** 3 + 7 * x4 ** 2 + 3 * x4 + 5 + bias
loss = torch.mean(x5)
print("loss =",loss)
loss.backward()
print("输出梯度信息:")
print(x0.grad)

控制台输出结果:

Windows PowerShell
版权所有 (C) Microsoft Corporation。保留所有权利。

尝试新的跨平台 PowerShell https://aka.ms/pscore6

加载个人及系统配置文件用了 901 毫秒。
(base) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> conda activate ssd4pytorch1_2_0
(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq>  & 'D:\Anaconda3\envs\ssd4pytorch1_2_0\python.exe' 'c:\Users\chenxuqi\.vscode\extensions\ms-python.python-2020.12.424452561\pythonFiles\lib\python\debugpy\launcher' '49739' '--' 'c:\Users\chenxuqi\Desktop\News4cxq\test4cxq\test25.py'
loss = tensor(9.6161e+08, grad_fn=<MeanBackward0>)
输出梯度信息:
tensor([[5.4391e+07, 1.7363e+07, 1.3316e+08, 3.6766e+05, 2.4903e+07],
        [5.1519e+08, 1.5781e+08, 3.6020e+05, 1.7468e+08, 1.2224e+08],
        [4.6100e+07, 9.8575e+07, 6.9595e+05, 2.4009e+07, 1.6252e+07]])
(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> 

3.使用copy_()的情况:x4 = x4.copy_(x2)或者x4.copy_(x2)
解释说明: 将x2的数据复制到x4,并且会修改计算图,使得反向传播自动计算梯度时,计算出x4的梯度后再继续前向计算x2的梯度.

import torch
torch.manual_seed(seed=20200910)
x0 = torch.randn(3, 5, requires_grad=True)
bias = torch.randn(3, 5, requires_grad=True)
x3 = torch.randn(3, 5, requires_grad=False)
x4 = torch.randn(3, 5, requires_grad=False)
x1 = 3 * x0 + 8
x2 = 4 * x1 ** 2 + 5 * x1 + 7

#
# x3 = x2.clone()
# x3 = x3.copy_(x2)
# x3 = x2.detach()
# x3 = x2.detach_()
# x3 = x2

#
# x4 = x2.clone()
# x4 = x4.copy_(x2)
# x4.copy_(x3)
x4.copy_(x2)
# x4 = x2.detach()
# x4 = x2.detach_()
# x4 = x2

#
x5 = 9 * x4 ** 3 + 7 * x4 ** 2 + 3 * x4 + 5 + bias
loss = torch.mean(x5)
print("loss =",loss)
loss.backward()
print("输出梯度信息:")
print(x0.grad)

控制台输出结果:

Windows PowerShell
版权所有 (C) Microsoft Corporation。保留所有权利。

尝试新的跨平台 PowerShell https://aka.ms/pscore6

加载个人及系统配置文件用了 914 毫秒。
(base) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> conda activate ssd4pytorch1_2_0
(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq>  & 'D:\Anaconda3\envs\ssd4pytorch1_2_0\python.exe' 'c:\Users\chenxuqi\.vscode\extensions\ms-python.python-2020.12.424452561\pythonFiles\lib\python\debugpy\launcher' '49845' '--' 'c:\Users\chenxuqi\Desktop\News4cxq\test4cxq\test25.py'
loss = tensor(9.6161e+08, grad_fn=<MeanBackward0>)
输出梯度信息:
tensor([[5.4391e+07, 1.7363e+07, 1.3316e+08, 3.6766e+05, 2.4903e+07],
        [5.1519e+08, 1.5781e+08, 3.6020e+05, 1.7468e+08, 1.2224e+08],
        [4.6100e+07, 9.8575e+07, 6.9595e+05, 2.4009e+07, 1.6252e+07]])
(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq>

注意: 如果将x4.copy_(x2)改成x4.copy_(x3),那么反向传播无法传递到x0,控制台输出的结果是None:

import torch
torch.manual_seed(seed=20200910)
x0 = torch.randn(3, 5, requires_grad=True)
bias = torch.randn(3, 5, requires_grad=True)
x3 = torch.randn(3, 5, requires_grad=False)
x4 = torch.randn(3, 5, requires_grad=False)
x1 = 3 * x0 + 8
x2 = 4 * x1 ** 2 + 5 * x1 + 7

#
# x3 = x2.clone()
# x3 = x3.copy_(x2)
# x3 = x2.detach()
# x3 = x2.detach_()
# x3 = x2

#
# x4 = x2.clone()
# x4 = x4.copy_(x2)
x4.copy_(x3)
# x4.copy_(x2)
# x4 = x2.detach()
# x4 = x2.detach_()
# x4 = x2

#
x5 = 9 * x4 ** 3 + 7 * x4 ** 2 + 3 * x4 + 5 + bias
loss = torch.mean(x5)
print("loss =",loss)
loss.backward()
print("输出梯度信息:")
print(x0.grad)
Windows PowerShell
版权所有 (C) Microsoft Corporation。保留所有权利。

尝试新的跨平台 PowerShell https://aka.ms/pscore6

加载个人及系统配置文件用了 873 毫秒。
(base) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> conda activate ssd4pytorch1_2_0
(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq>  & 'D:\Anaconda3\envs\ssd4pytorch1_2_0\python.exe' 'c:\Users\chenxuqi\.vscode\extensions\ms-python.python-2020.12.424452561\pythonFiles\lib\python\debugpy\launcher' '49870' '--' 'c:\Users\chenxuqi\Desktop\News4cxq\test4cxq\test25.py'
loss = tensor(16.7390, grad_fn=<MeanBackward0>)
输出梯度信息:
None
(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq>

4.使用detach()函数情况:x4 = x2.detach()
解释说明: 返回一个和原张量共享内存的新张量,两者的改动可以相互可见, 返回的新张量和计算图相互独立,即新张量和计算图不再关联,因此也无法 进行反向传播计算梯度.

import torch
torch.manual_seed(seed=20200910)
x0 = torch.randn(3, 5, requires_grad=True)
bias = torch.randn(3, 5, requires_grad=True)
x3 = torch.randn(3, 5, requires_grad=False)
x4 = torch.randn(3, 5, requires_grad=False)
x1 = 3 * x0 + 8
x2 = 4 * x1 ** 2 + 5 * x1 + 7

#
# x3 = x2.clone()
# x3 = x3.copy_(x2)
# x3 = x2.detach()
# x3 = x2.detach_()
# x3 = x2

#
# x4 = x2.clone()
# x4 = x4.copy_(x2)
# x4.copy_(x3)
# x4.copy_(x2)
x4 = x2.detach()
print('x4.requires_grad:',x4.requires_grad)
# x4 = x2.detach_()
# x4 = x2

#
x5 = 9 * x4 ** 3 + 7 * x4 ** 2 + 3 * x4 + 5 + bias
loss = torch.mean(x5)
print("loss =",loss)
loss.backward()
print("输出梯度信息:")
print(x0.grad)

控制台输出结果:

Windows PowerShell
版权所有 (C) Microsoft Corporation。保留所有权利。

尝试新的跨平台 PowerShell https://aka.ms/pscore6

加载个人及系统配置文件用了 975 毫秒。
(base) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> conda activate ssd4pytorch1_2_0
(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq>  & 'D:\Anaconda3\envs\ssd4pytorch1_2_0\python.exe' 'c:\Users\chenxuqi\.vscode\extensions\ms-python.python-2020.12.424452561\pythonFiles\lib\python\debugpy\launcher' '50013' '--' 'c:\Users\chenxuqi\Desktop\News4cxq\test4cxq\test25.py'
x4.requires_grad: False
loss = tensor(9.6161e+08, grad_fn=<MeanBackward0>)
输出梯度信息:
None
(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq>

5.使用detach_()的情况:x4 = x2.detach_()
解释说明: detach()的原地操作版本,功能和detach()类似.

import torch
torch.manual_seed(seed=20200910)
x0 = torch.randn(3, 5, requires_grad=True)
bias = torch.randn(3, 5, requires_grad=True)
x3 = torch.randn(3, 5, requires_grad=False)
x4 = torch.randn(3, 5, requires_grad=False)
x1 = 3 * x0 + 8
x2 = 4 * x1 ** 2 + 5 * x1 + 7

#
# x3 = x2.clone()
# x3 = x3.copy_(x2)
# x3 = x2.detach()
# x3 = x2.detach_()
# x3 = x2

#
# x4 = x2.clone()
# x4 = x4.copy_(x2)
# x4.copy_(x3)
# x4.copy_(x2)
# x4 = x2.detach()
# print('x4.requires_grad:',x4.requires_grad)
x4 = x2.detach_()
print('x4.requires_grad:',x4.requires_grad)
# x4 = x2

#
x5 = 9 * x4 ** 3 + 7 * x4 ** 2 + 3 * x4 + 5 + bias
loss = torch.mean(x5)
print("loss =",loss)
loss.backward()
print("输出梯度信息:")
print(x0.grad)

控制台输出结果:

Windows PowerShell
版权所有 (C) Microsoft Corporation。保留所有权利。

尝试新的跨平台 PowerShell https://aka.ms/pscore6

加载个人及系统配置文件用了 915 毫秒。
(base) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> conda activate ssd4pytorch1_2_0
(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq>  & 'D:\Anaconda3\envs\ssd4pytorch1_2_0\python.exe' 'c:\Users\chenxuqi\.vscode\extensions\ms-python.python-2020.12.424452561\pythonFiles\lib\python\debugpy\launcher' '50131' '--' 'c:\Users\chenxuqi\Desktop\News4cxq\test4cxq\test25.py'
x4.requires_grad: False
loss = tensor(9.6161e+08, grad_fn=<MeanBackward0>)
输出梯度信息:
None
(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq>
  • 32
    点赞
  • 73
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
PyTorchclone函数用于创建一个张量的副本。通过clone函数创建的副本张量与原始张量具有相同的值,但是它们是不同的张量对象。这意味着副本张量可以独立于原始张量进行操作,而不会影响原始张量的梯度计算。 根据引用\[1\]和\[2\]的例子,当使用clone函数创建副本张量时,副本张量会保留梯度信息,可以进行梯度回溯。这意味着副本张量的梯度会传递给原始张量,并与原始张量的梯度进行叠加。 然而,根据引用\[3\]的例子,clone函数本身不会保存副本张量的梯度信息。当对副本张量进行梯度计算时,副本张量的梯度为None。只有原始张量会保存梯度信息。 因此,总结来说,clone函数PyTorch用于创建一个张量的副本,副本张量与原始张量具有相同的值,但是它们是不同的张量对象。副本张量可以进行独立的操作,但是在梯度计算,副本张量的梯度会传递给原始张量进行叠加,而副本张量本身不保存梯度信息。 #### 引用[.reference_title] - *1* *2* [【python基础】PyTorchclone()、detach()](https://blog.csdn.net/dujuancao11/article/details/121563226)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v91^control,239^v3^insert_chatgpt"}} ] [.reference_item] - *3* [pytorch的.clone() 和 .detach()复制变量、共享内存变量](https://blog.csdn.net/weixin_44503976/article/details/126631909)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v91^control,239^v3^insert_chatgpt"}} ] [.reference_item] [ .reference_list ]

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值