学习笔记《Grokking Deep Learning》

第三章 神经网络预测导论：前向传播

一个简单的神经网络

#网络代码
weight = 0.1
def neural_network(inpiut, weight):
	prediction = input * weight
	return prediction
#如何用训练好的神经网络进行预测
number_of_toes = [8.5, 9.5, 10, 9]

input = number_of_toes[0]

pred = neural_network(input, weight)
print(pred)

使用多个输入进行预测

多输入，单输出

import numpy as np

weights = np.array([0.1, 0.2, 0])

def neural_network(input, weights):
    pred = input.dot(weights)
    return pred

toes =  np.array([8.5, 9.5, 9.9, 10])
wlrec =  np.array([0.65, 0.8, 0.8, 0.9])
nfans =  np.array([1.2, 1.3, 0.5, 1.0])
input = np.array([toes[0], wlrec[0], nfans[0]])

perd = neural_network(input, weights)
print(perd)

单输入，多输出

# 单输入多输出
# 网络
def ele_mul(number, vector):
    output = [0, 0, 0]
    assert(len(output) == len(vector))
    for i in range(len(output)):
        output[i] = number * vector[i]
    return output

def neural_network(input, weights):
    pred = ele_mul(input, weights)
    return pred

# 数据
wlrec = [0.65, 0.8, 0.8, 0.9]
weights = [0.3, 0.2, 0.9]
input = wlrec[0]
pred = neural_network(input, weights)
print(pred)

多输入，多输出

def w_sum(a, b):
    assert(len(a) == len(b))
    output = 0
    for i in range(len(a)):
        output += (a[i] * b[i])
    return output

def vect_mat_mul(vect, matrix):
    assert(len(vect) == len(matrix))
    output = [0, 0, 0]
    for i in range(len(vect)):
        output[i] = w_sum(vect, matrix[i]) 
    return output

def neural_network(input, weights):
    pred = vect_mat_mul(input, weights)
    return pred

weights = [[0.1, 0.1, -0.3],
            [0.1, 0.2, 0.0],
            [0.0, 1.3, 0.1]]

toes = [8.5, 9.5, 9.9, 10]
wlrec = [0.65, 0.8, 0.8, 0.9]
nfans = [1.2, 1.3, 0.5, 1.0]

input = [toes[0], wlrec[0], nfans[0]]

pred = neural_network(input, weights)
print(pred)

用预测结果进一步预测

神经网络可以堆叠

import numpy as np

ih_wgh = np.array([[0.1, 0.1, -0.1],
                    [-0.1, 0.1, 0.9],
                    [0.1, 0.4, 0.1]])

hp_wgh = np.array([[0.3, 1.1, -0.3],
                    [0.1, 0.2, 0.0],
                    [0.0, 1.3, 0.1]])

weights = [ih_wgh, hp_wgh]

def neural_network(input, weights):
    hid = input.dot(weights[0])
    pred = hid.dot(weights[1])
    return pred

toes =  np.array([8.5, 9.5, 9.9, 10])
wlrec =  np.array([0.65, 0.8, 0.8, 0.9])
nfans =  np.array([1.2, 1.3, 0.5, 1.0])

input = np.array([toes[0], wlrec[0], nfans[0]])

pred = neural_network(input, weights)
print(pred)

第四章 梯度下降

1.冷热法

优点：简单明了
缺点：1.效率低下， 2、有时准确预测出目标是不可能的

# 冷热法

weight = 0.5
input = 0.5
goal_prediction = 0.8
#学习率?
step_amount = 0.001 

for _ in range(1101):
    prediction = input * weight
    error = (prediction - goal_prediction) ** 2

    print("Error:" + str(error) + " Prediction:" + str(prediction))

    up_prediction = input * (weight + step_amount)
    up_error = (up_prediction - goal_prediction) ** 2

    down_prediction = input * (weight - step_amount)
    down_error = (down_prediction - goal_prediction) ** 2

    if(down_error < error):
        weight -= step_amount
    if(up_error < error):
        weight += step_amount

Error:4.000000000130569e-06 Prediction:0.7979999999999674
Error:2.2500000000980924e-06 Prediction:0.7984999999999673
Error:1.000000000065505e-06 Prediction:0.7989999999999673
Error:2.5000000003280753e-07 Prediction:0.7994999999999672
Error:1.0799505792475652e-27 Prediction:0.7999999999999672

2.基于误差调节权重

weight = 0.5
goal_pred = 0.8
input = 0.5

for _ in range(20):
    pred = input * weight
    error = (pred - goal_pred) ** 2
    # 有正负，预测比实际大，为 + ；预测比实际小， 为 -
    direction_and_amount = (pred - goal_pred) * input
    weight -= direction_and_amount
    print("Error:" + str(error) + " Prediction:" + str(pred))

Error:0.000303525861459885 Prediction:0.7825780063867569
Error:0.00017073329707118678 Prediction:0.7869335047900676
Error:9.603747960254256e-05 Prediction:0.7902001285925507
Error:5.402108227642978e-05 Prediction:0.7926500964444131
Error:3.038685878049206e-05 Prediction:0.7944875723333098
Error:1.7092608064027242e-05 Prediction:0.7958656792499823
Error:9.614592036015323e-06 Prediction:0.7968992594374867
Error:5.408208020258491e-06 Prediction:0.7976744445781151

3.破坏梯度下降(当输入值过大)

把input改为2

weight = 0.5
goal_pred = 0.8
input = 2

for _ in range(20):
    pred = input * weight
    error = (pred - goal_pred) ** 2
    # 有正负，预测比实际大，为 + ；预测比实际小， 为 -
    direction_and_amount = (pred - goal_pred) * input
    weight -= direction_and_amount
    print("Error:" + str(error) + " Prediction:" + str(pred))

Error:101674633133.15994 Prediction:-318863.79999999993
Error:915071698198.4395 Prediction:956594.5999999997
Error:8235645283785.954 Prediction:-2869780.599999999
Error:74120807554073.56 Prediction:8609344.999999996
Error:667087267986662.1 Prediction:-25828031.799999986
Error:6003785411879960.0 Prediction:77484098.59999996
Error:5.403406870691965e+16 Prediction:-232452292.5999999

得到的结果没有意义
对权重的每次更新都会造成过度的修正

引入alpha 学习率?

alpha 过大: 权重更新步幅大, 容易在最低点跳过去, 不易取到最小值
alpha 过小: 权重更新步幅小,每次更新变化小,需要大量迭代计算,效率低下

alpha的取值没有明文规定, 一般取数量级: 10, 1, 0.1, 0.01 …

#引入 alpha
weight = 0.5
goal_pred = 0.8
input = 2
alpha = 0.1

for _ in range(20):
    pred = input * weight
    error = (pred - goal_pred) ** 2
    direction_and_amount = (pred - goal_pred) * input
    weight -= direction_and_amount * alpha
    print("Error:" + str(error) + " Prediction:" + str(pred))

Error:1.1284439629823931e-05 Prediction:0.803359232
Error:4.062398266736526e-06 Prediction:0.8020155392
Error:1.4624633760252567e-06 Prediction:0.8012093235200001
Error:5.264868153690924e-07 Prediction:0.8007255941120001
Error:1.8953525353291194e-07 Prediction:0.8004353564672001
Error:6.82326912718715e-08 Prediction:0.8002612138803201
Error:2.456376885786678e-08 Prediction:0.8001567283281921
Error:8.842956788836216e-09 Prediction:0.8000940369969153
Error:3.1834644439835434e-09 Prediction:0.8000564221981492
Error:1.1460471998340758e-09 Prediction:0.8000338533188895
Error:4.125769919393652e-10 Prediction:0.8000203119913337
Error:1.485277170987127e-10 Prediction:0.8000121871948003

第五章 通用梯度下降：一次学习多个权重

1.多输入，单输出，一次学习多个权重的梯度下降

delta

delta = pred - true

与误差不同,考虑到误差为正，且专注大的误差，放弃小的误差

error = (pred - true) ** 2

delta 用于衡量你所希望的当前节点变化，以便完美地预测结果

weight_delta

weights_deltas = ele_mul(delta, input)

代表基于导数的对权重移动方向和数量的估计，目标是降低 detal。

# 多输入，单输出梯度下降

def neural_network(input, weights):
    out = 0
    for i in range(len(input)):
        out += (input[i] * weights[i])
    return out 

def ele_mul(scalar, vector):
    out = [0, 0, 0]
    for i in range(len(out)):
        out[i] = vector[i] * scalar
    return out

toes = [8.5, 9.5, 9.9, 10]
wlrec = [0.65, 0.8, 0.8, 0.9]
nfans = [1.2, 1.3, 0.5, 1.0]

win_or_loss_binary = [1, 1, 0, 1]
true = win_or_loss_binary[0]

alpha = 0.01
weights = [0.1, 0.2, -0.1]
input = [toes[0], wlrec[0], nfans[0]]

for iter in range(3):
    pred = neural_network(input, weights)

    error = (pred - true) ** 2
    delta = pred - true

    weights_deltas = ele_mul(delta, input)
    print("Error:" + str(error) + "     Prediction:" + str(pred) + "    Delta:" + str(delta)
            + "   Weight_delta:" + str(weights_deltas))
    for i in range(len(weights)):
        weights[i] -= weights_deltas[i] * alpha

Error:0.01959999999999997     Prediction:0.8600000000000001    Delta:-0.1399999999999999   Weight_delta:[-1.189999999999999, -0.09099999999999994, -0.16799999999999987]
Error:0.0013135188062500048     Prediction:0.9637574999999999    Delta:-0.036242500000000066   Weight_delta:[-0.30806125000000056, -0.023557625000000044, -0.04349100000000008]
Error:8.802712522307997e-05     Prediction:0.9906177228125002    Delta:-0.009382277187499843   Weight_delta:[-0.07974935609374867, -0.006098480171874899, -0.011258732624999811]

2.单项权重冻结

冻结toes[0]的权重——将其对应的weights_delta设为0

# 单项权重冻结

def neural_network(input, weights):
    out = 0
    for i in range(len(input)):
        out += (input[i] * weights