PyTorch: Tensors and autograd 2020.04.26

PyTorch: Tensors and autograd 2020.04.26

w1=torch.randn(D_in,H,device=device,dtype=dtype,requires_grad=True)
w2=torch.randn(H,D_out,device=device,dtype=dtype,requires_grad=True)
#因为w是需要进行梯度下降更新的，所以将其设为requires_grad=True

for i in range(500):
y_pred=x.mm(w1).clamp(min=0).mm(w2)
loss=(y_pred-y).pow(2).sum()
if i%100==99:
print(i,loss.item())
loss.backward()
with torch.no_grad():
w1-=learning_ratew1.grad
w2-=learning_ratew2.grad
w1.grad.zero_()
w2.grad.zero_()

注释：红笔部分均为没有想到的方法。需要记忆。

nn module
PyTorch: nn

A Module receives input Tensors and computes output Tensors, but may also hold internal state such as Tensors containing learnable parameters. The nn package also defines a set of useful loss functions that are commonly used when training neural networks. The nn package defines a set of Modules, which are roughly equivalent to neural network layers.

model=torch.nn.Sequential(
torch.nn.Linear(D_in,H),

一般形如y=kx+b（k，b是常数，k≠0）,就相当于前向传播的权重和输入相乘加上偏置。

torch.nn.ReLU(), torch.nn.Linear(H,D_out),)

在进行迭代前把学习率和损失函数，网络结构都定义好
for t in range(500):进行迭代，得带过程首先
(1)计算模型所产生值，前向传播的过程。
(2)计算loss
(3)在运行后向遍历之前使梯度为零
(4)定义loss可以反向传播
(5)进行反向传播，进行参数更新
for i in range(50000):
y_pred=model(x)
loss=loss_fn(y_pred,y)
if i%1000==99:
print(i,loss.item())
model.zero_grad()
loss.backward()
with torch.no_grad():
for param in model.parameters():
param-=learning_rate*param.grad

PyTorch: optim

在进行迭代前把学习率和损失函数，网络结构都定义好。再加入优化器后只需要把下面的语句加在迭代前即可。
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

if t % 100 == 99:
print(t, loss.item())
optimizer.zero_grad()
#这里和上边相比，这里直接用优化器进行清零操作。
loss.backward()
#和上边无差别
optimizer.step()
#用优化器更新参数。