pytorch学习笔记（二）用numpy搭建一个简单的网络

在深度学习框架没出来之前，numpy是用来做机器学习的矩阵计算和梯度计算的，一直以来对与神经网络的权值更新机制不太了解，只知道是w = w -learningrate*梯度这种形式，在pytorch的教程中看到基于numpy的demo，特在此记录一下。

import numpy as np
"""use numpy"""
N,D_in,H,D_out = 64,1000,100,10
x = np.random.randn(N,D_in)
y = np.random.randn(N,D_out)
w1 = np.random.randn(D_in,H)
w2 = np.random.randn(H,D_out)
lr = 1e-6
for t in range(500):
        """calculate the foward"""
        h = x.dot(w1)
        h_relu = np.maximum(h,0)
        y_pred = h_relu.dot(w2)
        """calculate the loss"""
        loss = np.square(y_pred-y).sum()
        print ("epoch",t,"loss",loss)
        grad_y_pred = 2.0*(y_pred -y)
        grad_w2 = h_relu.T.dot(grad_y_pred)
        grad_h_relu = grad_y_pred.dot(w2.T)
        grad_h = grad_h_relu.copy()
        grad_h[h<0] = 0
        grad_w1 = x.T.dot(grad_h)
        w1 -= lr*grad_w1
        w2 -= lr*grad_w2

结果如下

epoch 0 loss 40279650.670588285
epoch 1 loss 38618569.4854403
epoch 2 loss 41338785.80719532
epoch 3 loss 38971053.78169297
epoch 4 loss 28903296.775348
epoch 5 loss 16185244.67426411
epoch 6 loss 7669148.284338362
epoch 7 loss 3648781.9787034444
epoch 8 loss 2027601.8378013337
epoch 9 loss 1347429.182238786
epoch 10 loss 1014348.726412890

500次迭代的结果，可以看见loss已经非常小了。

epoch 480 loss 3.2881478517825267e-06
epoch 481 loss 3.1424578084813636e-06
epoch 482 loss 3.0032340935108157e-06
epoch 483 loss 2.8702000764324365e-06
epoch 484 loss 2.7430897582326752e-06
epoch 485 loss 2.621605112785752e-06
epoch 486 loss 2.5055023387805042e-06
epoch 487 loss 2.394552895715606e-06
epoch 488 loss 2.2885253240073905e-06
epoch 489 loss 2.1872447497125805e-06
epoch 490 loss 2.0904326320463472e-06
epoch 491 loss 1.9978954242027575e-06
epoch 492 loss 1.9094639843526256e-06
epoch 493 loss 1.8249516150514206e-06
epoch 494 loss 1.7442084704744092e-06
epoch 495 loss 1.667025266203989e-06
epoch 496 loss 1.593263262804167e-06
epoch 497 loss 1.5227728074326867e-06
epoch 498 loss 1.4554065539966364e-06
epoch 499 loss 1.3910406563476352e-06