第四讲反向传播

1、第四讲反向传播的笔记和源码

B站 刘二大人 教学视频 传送门： 反向传播

①前沿代码提示部份：

 forward函数的返回值也是Tensor，loss函数的返回值也是Tensor

②本算法中反向传播主要体现在，l.backward()。调用该方法后w.grad由None更新为Tensor类型，且w.grad.data的值用于后续w.data的更新。

 ③ l.backward()会把计算图中所有需要梯度(grad)的地方都会求出来，之后把梯度都存在对应的待求的参数中，最终计算图被释放。取tensor中的data是不会构建计算图的

---

源码：模型为：y=w*x,使用pytorch 实现反向传播的代码如下，

import torch

x_data=[1.0,2.0,3.0]
y_data=[2.0,4.0,6.0]

w=torch.Tensor([1.0])# w的初始值为1.0
w.requires_grad=True #需要计算梯度

# 前馈
def forward(x):
    return x*w #w是一个Tensor
# 构建计算图
def loss(x,y):
    y_pred=forward(x)
    return (y_pred-y)**2

print("predict (before training)",4,forward(4).item())

for epoch in range(100):
    for x,y in zip(x_data,y_data):
        l=loss(x,y) #l是一个张良，tensor主要是在建立计算图forward
        l.backward()
        print('\tgrad:',x,y,w.grad.item())
        w.data=w.data-0.01 *w.grad.data

        w.grad.data.zero_() # 张量的梯度清0 进行更新

 
    print("progress:",epoch,l.item()) #取出loss 使用l.item 不要直接使用l(l是tensor会构建计算图)
print("predict (after training)",4,forward(4).item())

输出结果：

predict (before training) 4 4.0
    grad: 1.0 2.0 -2.0
    grad: 2.0 4.0 -7.840000152587891
    grad: 3.0 6.0 -16.228801727294922
progress: 0 7.315943717956543
    grad: 1.0 2.0 -1.478623867034912
    grad: 2.0 4.0 -5.796205520629883
    grad: 3.0 6.0 -11.998146057128906
progress: 1 3.9987640380859375
    grad: 1.0 2.0 -1.0931644439697266
    grad: 2.0 4.0 -4.285204887390137
    grad: 3.0 6.0 -8.870372772216797

.............
progress: 94 9.094947017729282e-13
    grad: 1.0 2.0 -7.152557373046875e-07
    grad: 2.0 4.0 -2.86102294921875e-06
    grad: 3.0 6.0 -5.7220458984375e-06
progress: 95 9.094947017729282e-13
    grad: 1.0 2.0 -7.152557373046875e-07
    grad: 2.0 4.0 -2.86102294921875e-06
    grad: 3.0 6.0 -5.7220458984375e-06
progress: 96 9.094947017729282e-13
    grad: 1.0 2.0 -7.152557373046875e-07
    grad: 2.0 4.0 -2.86102294921875e-06
    grad: 3.0 6.0 -5.7220458984375e-06
progress: 97 9.094947017729282e-13
    grad: 1.0 2.0 -7.152557373046875e-07
    grad: 2.0 4.0 -2.86102294921875e-06
    grad: 3.0 6.0 -5.7220458984375e-06
progress: 98 9.094947017729282e-13
    grad: 1.0 2.0 -7.152557373046875e-07
    grad: 2.0 4.0 -2.86102294921875e-06
    grad: 3.0 6.0 -5.7220458984375e-06
progress: 99 9.094947017729282e-13
predict (after training) 4 7.999998569488525

进程已结束，退出代码为 0

---

2、补充知识：来识别一下torch下面的某些参数类型

import torch
b=torch.tensor([1.0])
#也可以a.requires_grad()
b.requires_grad=True
print(b)
print(b.data)
print(b.type()) #b是tensor
print(b.data.type()) # b.data的类型是tensor
print(b.grad)
print(type(b.grad))

结果为

tensor([1.], requires_grad=True)
tensor([1.])
torch.FloatTensor
torch.FloatTensor
None
<class 'NoneType'>

---

3、第四讲反向传播作业：

1、手动推导线性模型y=w*x与y=w*x+b与y=w1*x^2+w2*x+b，损失函数loss=(ŷ-y)²下，当数据集x=?,y=?的时候，反向传播的过程。且用pytorch实现(y=w1*x^2+w2*x+b    损失函数loss=(ŷ-y)²)反向传播的代码。

---

使用pytorch实现二次函数反向传播代码实现：

import torch

x_data=[1.0,2.0,3.0]
y_data=[2.0,4.0,6.0]

w1=torch.Tensor([1.0])# w的初始值为1.0
w1.requires_grad=True #需要计算梯度
w2=torch.Tensor([1.0])
w2.requires_grad=True
b=torch.Tensor([1.0])
b.requires_grad=True

# 前馈
def forward(x):
    return w1*x**2+w2*x+b #w是一个Tensor
# 构建计算图
def loss(x,y):
    y_pred=forward(x)
    return (y_pred-y)**2

print("predict (before training)",4,forward(4).item())

for epoch in range(100):
    for x,y in zip(x_data,y_data):
        l=loss(x,y) #l是一个张量，tensor主要是在建立计算图forward
        l.backward()
        print('\tgrad:',x,y,w1.grad.item(),w2.grad.item(),b.grad.item())
        w1.data=w1.data-0.01 *w1.grad.data
        w2.data=w2.data-0.01 *w2.grad.data
        b.data=b.data-0.01 *b.grad.data

        w1.grad.data.zero_() # 张量的梯度清0 进行更新
        w2.grad.data.zero_()
        b.grad.data.zero_()

    print("progress:",epoch,l.item()) #取出loss 使用l.item 不要直接使用l(l是tensor会构建计算图)
print("predict (after training)",4,forward(4).item())

---

运行结果：

predict (before training) 4 21.0
    grad: 1.0 2.0 2.0 2.0 2.0
    grad: 2.0 4.0 22.880001068115234 11.440000534057617 5.720000267028809
    grad: 3.0 6.0 77.04720306396484 25.682401657104492 8.560800552368164
progress: 0 18.321826934814453
    grad: 1.0 2.0 -1.1466078758239746 -1.1466078758239746 -1.1466078758239746
    grad: 2.0 4.0 -15.536651611328125 -7.7683258056640625 -3.8841629028320312
    grad: 3.0 6.0 -30.432214736938477 -10.144071578979492 -3.381357192993164
progress: 1 2.858394145965576
    grad: 1.0 2.0 0.3451242446899414 0.3451242446899414 0.3451242446899414
    grad: 2.0 4.0 2.4273414611816406 1.2136707305908203 0.6068353652954102
    grad: 3.0 6.0 19.449920654296875 6.483306884765625 2.161102294921875

........
progress: 66 0.007711691781878471
    grad: 1.0 2.0 0.29669761657714844 0.29669761657714844 0.29669761657714844
    grad: 2.0 4.0 -1.841745376586914 -0.920872688293457 -0.4604363441467285
    grad: 3.0 6.0 1.5733451843261719 0.5244483947753906 0.17481613159179688
progress: 67 0.007640169933438301
    grad: 1.0 2.0 0.29790449142456055 0.29790449142456055 0.29790449142456055
    grad: 2.0 4.0 -1.8364067077636719 -0.9182033538818359 -0.45910167694091797
    grad: 3.0 6.0 1.5662040710449219 0.5220680236816406 0.17402267456054688
progress: 68 0.007570972666144371
    grad: 1.0 2.0 0.2990589141845703 0.2990589141845703 0.2990589141845703
    grad: 2.0 4.0 -1.8312263488769531 -0.9156131744384766 -0.4578065872192383
    grad: 3.0 6.0 1.5593376159667969 0.5197792053222656 0.17325973510742188
progress: 69 0.007504733745008707
    grad: 1.0 2.0 0.30016040802001953 0.30016040802001953 0.30016040802001953
    grad: 2.0 4.0 -1.8262138366699219 -0.9131069183349609 -0.45655345916748047
    grad: 3.0 6.0 1.552694320678711 0.5175647735595703 0.17252159118652344

.......
progress: 77 0.00705681974068284
    grad: 1.0 2.0 0.3073387145996094 0.3073387145996094 0.3073387145996094
    grad: 2.0 4.0 -1.791269302368164 -0.895634651184082 -0.447817325592041
    grad: 3.0 6.0 1.5070152282714844 0.5023384094238281 0.16744613647460938
progress: 78 0.007009552326053381
    grad: 1.0 2.0 0.3080568313598633 0.3080568313598633 0.3080568313598633
    grad: 2.0 4.0 -1.7874603271484375 -0.8937301635742188 -0.4468650817871094
    grad: 3.0 6.0 1.502131462097168 0.5007104873657227 0.16690349578857422
progress: 79 0.006964194122701883
    grad: 1.0 2.0 0.30873966217041016 0.30873966217041016 0.30873966217041016
    grad: 2.0 4.0 -1.7837696075439453 -0.8918848037719727 -0.44594240188598633
    grad: 3.0 6.0 1.4973936080932617 0.4991312026977539 0.16637706756591797
progress: 80 0.006920332089066505
    grad: 1.0 2.0 0.309389591217041 0.309389591217041 0.309389591217041
    grad: 2.0 4.0 -1.780181884765625 -0.8900909423828125 -0.44504547119140625
    grad: 3.0 6.0 1.492818832397461 0.4976062774658203 0.16586875915527344

.......
progress: 95 0.006416172254830599
    grad: 1.0 2.0 0.3158855438232422 0.3158855438232422 0.3158855438232422
    grad: 2.0 4.0 -1.7366409301757812 -0.8683204650878906 -0.4341602325439453
    grad: 3.0 6.0 1.4389429092407227 0.4796476364135742 0.1598825454711914
progress: 96 0.006390606984496117
    grad: 1.0 2.0 0.3161449432373047 0.3161449432373047 0.3161449432373047
    grad: 2.0 4.0 -1.7342891693115234 -0.8671445846557617 -0.43357229232788086
    grad: 3.0 6.0 1.436136245727539 0.4787120819091797 0.15957069396972656
progress: 97 0.0063657015562057495
    grad: 1.0 2.0 0.3163881301879883 0.3163881301879883 0.3163881301879883
    grad: 2.0 4.0 -1.7319889068603516 -0.8659944534301758 -0.4329972267150879
    grad: 3.0 6.0 1.4334239959716797 0.47780799865722656 0.1592693328857422
progress: 98 0.0063416799530386925
    grad: 1.0 2.0 0.31661415100097656 0.31661415100097656 0.31661415100097656
    grad: 2.0 4.0 -1.7297439575195312 -0.8648719787597656 -0.4324359893798828
    grad: 3.0 6.0 1.4307546615600586 0.47691822052001953 0.15897274017333984
progress: 99 0.00631808303296566
predict (after training) 4 8.544171333312988

进程已结束，退出代码为 0

---

Ending, 如有错误敬请指正，感谢！