吴恩达深度学习 | (9) 改善深层神经网络专项课程第一周编程作业

吴恩达深度学习专项课程的所有实验均采用iPython Notebooks实现，不熟悉的朋友可以提前使用一下Notebooks。本周包含三个实验，这三个实验都基于一个3层神经网络模型(模型已经帮助你实现好了）进行演示。实验1要求实现三种不同的权重参数初始化方式，通过对其效果的比较，说明参数初始化的重要性，选择一个最好的初始化方式；实验2要求实现正则化方法，包括L2正则化和dropout，比较使用/不使用正则化模型的效果，说明正则化在解决模型过拟合方面的重要性；实验3要求实现梯度检验，来检查反向传播计算的梯度，确保反向传播实现的正确性。

 

目录

一、实验1:初始化

1.实验综述

2.导入必要的包

3.神经网络模型

3.Zero 初始化

4.随机初始化

5.He 初始化

6.结论

二、实验2:正则化

1.实验综述

2.导入必要的包

3.不带正则化的模型

4.L2正则化

5.Dropout

6.结论

三、实验3:梯度检查

1.实验综述

2.导入必要的包

3.梯度检验时怎么工作的？

四、实验完整代码

---

一、实验1:初始化

1.实验综述

2.导入必要的包

import numpy as np
import matplotlib.pyplot as plt
import sklearn
import sklearn.datasets
#init_utils.py 中定义了3层神经网络结构相关的函数实现 以及数据集的加载函数 绘制决策边界的函数
from init_utils import sigmoid, relu, compute_cost, forward_propagation, backward_propagation
from init_utils import update_parameters, predict, load_dataset, plot_decision_boundary, predict_dec

%matplotlib inline
#设置绘图的默认风格
plt.rcParams['figure.figsize'] = (7.0, 4.0) # 默认大小
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray' #默认色调

#加载平面数据集
train_X, train_Y, test_X, test_Y = load_dataset()

• 待分类平面数据集可视化效果

• 实验任务

3.神经网络模型

• 定义神经网络模型

#神经网络模型
def model(X, Y, learning_rate = 0.01, num_iterations = 15000, print_cost=False,initialization="he"):
    '''
    训练网络
    
    参数：
    X：训练集样本特征矩阵(n_x,m_train)
    Y:训练集样本标签 0/1 (1,m_train)
    learning_rate:学习率
    num_iterations:梯度下降法迭代次数
    print_cost:为True时，每1000次迭代打印一次cost
    initialization:选择权重参数初始化方式 "zeros","random","he"
    
    返回：
    parameters：训练好的参数 字典形式
        parameters["W" + str(l)] = ... 
        parameters["b" + str(l)] = ...
    '''
    
    np.random.seed(1)
    costs = [] #存储每1000次前向传播计算的代价
    layers_dims = [X.shape[0],10,5,1] #非定义3层神经网络各层的单元数
    
    #初始化方式
    if initialization == "zeros":
        parameters = initialize_parameters_zeros(layers_dims)
    elif initialization == "random":
        parameters = initialize_parameters_random(layers_dims)
    elif initialization == "he":
        parameters = initialize_parameters_he(layers_dims)
    
    #梯度下降迭代
    for i in range(num_iterations):
        
        #前向传播
        a3,caches = forward_propagation(X,parameters)
        
        #计算代价
        cost = compute_cost(a3,Y)
        
        #反向传播计算梯度
        grads = backward_propagation(X,Y,caches)
        
        #更新参数
        parameters = update_parameters(parameters,grads,learning_rate)
        
        #每100次迭代打印一次代价 并存储
        if print_cost and i%1000==0:
            print("Cost after iteration {}: {}".format(i, np.squeeze(cost)))
            costs.append(cost)
            
    #绘制代价对迭代次数的变化曲线
    plt.plot(np.squeeze(costs))
    plt.ylabel('cost')
    plt.xlabel('iterations (per hundreds)')
    plt.title("Learning rate =" + str(learning_rate))
    plt.show()
    
    return parameters
        

3.Zero 初始化

• 代码实现 

def initialize_parameters_zeros(layers_dims):
    """
    参数：
    layer_dims:网络每一层的单元数
    
    返回：
    parameters：初始化后的参数 字典形式
       "W1", "b1", ..., "WL", "bL":
                    Wl -- weight matrix of shape (layer_dims[l], layer_dims[l-1])
                    bl -- bias vector of shape (layer_dims[l], 1)
    """
    parameters = {}
    L = len(layers_dims) #神经网络层数+1
    
    for l in range(1,L): #1～L-1
        parameters["W"+str(l)] = np.zeros((layers_dims[l],layers_dims[l-1]))
        parameters["b"+str(l)] = np.zeros((layers_dims[l],1))
    
    return parameters

• 测试代码

parameters = initialize_parameters_zeros([3,2,1])
print("W1 = " + str(parameters["W1"]))
print("b1 = " + str(parameters["b1"]))
print("W2 = " + str(parameters["W2"]))
print("b2 = " + str(parameters["b2"]))

• 训练模型

使用0初始化方式，训练神经网络模型，15000次梯度下降迭代

parameters = model(train_X, train_Y,print_cost=True,initialization = "zeros")
print ("On the train set:")
predictions_train = predict(train_X, train_Y, parameters)
print ("On the test set:")
predictions_test = predict(test_X, test_Y, parameters)

此时模型的表现非常差，代价并没有减小。算法表现可能还不如瞎猜。

• 查看模型在训练/测试集上的具体预测值

print ("predictions_train = " + str(predictions_train))
print ("predictions_test = " + str(predictions_test))

模型的预测结果全为0.

• 绘制决策边界

plt.title("Model with Zeros initialization")
axes = plt.gca()
axes.set_xlim([-1.5,1.5])
axes.set_ylim([-1.5,1.5])
plot_decision_boundary(lambda x: predict_dec(parameters,x.T), train_X, np.squeeze(train_Y))

• 总结

4.随机初始化

• 代码实现

def initialize_parameters_random(layers_dims):
    
    """
    参数：
    layer_dims:网络每一层的单元数
    
    返回：
    parameters：初始化后的参数 字典形式
       "W1", "b1", ..., "WL", "bL":
                    Wl -- weight matrix of shape (layer_dims[l], layer_dims[l-1])
                    bl -- bias vector of shape (layer_dims[l], 1)
    """
    np.random.seed(3)
    parameters = {}
    L = len(layers_dims) #神经网络层数+1
    
    for l in range(1,L): #1～L-1
        parameters["W"+str(l)] = np.random.randn(layers_dims[l],layers_dims[l-1])*10
        parameters["b"+str(l)] = np.zeros((layers_dims[l],1))
    return parameters

• 测试代码

parameters = initialize_parameters_random([3, 2, 1])
print("W1 = " + str(parameters["W1"]))
print("b1 = " + str(parameters["b1"]))

print("W2 = " + str(parameters["W2"]))
print("b2 = " + str(parameters["b2"]))

• 训练模型

使用随机初始化方式，训练神经网络模型，15000次梯度下降迭代

parameters = model(train_X, train_Y,print_cost=True,initialization = "random")
print ("On the train set:")
predictions_train = predict(train_X, train_Y, parameters)
print ("On the test set:")
predictions_test = predict(test_X, test_Y, parameters)

• 查看模型具体的预测值

print (predictions_train)
print (predictions_test)

• 绘制决策边界

plt.title("Model with large random initialization")
axes = plt.gca()
axes.set_xlim([-1.5,1.5])
axes.set_ylim([-1.5,1.5])
plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, np.squeeze(train_Y))

• 总结

5.He 初始化

• 代码实现

def initialize_parameters_he(layers_dims):
    
    """
    参数：
    layer_dims:网络每一层的单元数
    
    返回：
    parameters：初始化后的参数 字典形式
       "W1", "b1", ..., "WL", "bL":
                    Wl -- weight matrix of shape (layer_dims[l], layer_dims[l-1])
                    bl -- bias vector of shape (layer_dims[l], 1)
    """
    np.random.seed(3)
    parameters = {}
    L = len(layers_dims) #神经网络层数+1
    
    for l in range(1,L): #1～L-1
        parameters["W"+str(l)] = np.random.randn(layers_dims[l],layers_dims[l-1])*np.sqrt(2./layers_dims[l-1])
        parameters["b"+str(l)] = np.zeros((layers_dims[l],1))
    return parameters

• 测试代码

parameters = initialize_parameters_he([2, 4, 1])
print("W1 = " + str(parameters["W1"]))
print("b1 = " + str(parameters["b1"]))
print("W2 = " + str(parameters["W2"]))
print("b2 = " + str(parameters["b2"]))

• 训练模型

使用He初始化方式，训练神经网络模型，15000次梯度下降迭代。

parameters = model(train_X, train_Y,print_cost=True, initialization = "he")
print ("On the train set:")
predictions_train = predict(train_X, train_Y, parameters)
print ("On the test set:")
predictions_test = predict(test_X, test_Y, parameters)

• 绘制决策边界

plt.title("Model with He initialization")
axes = plt.gca()
axes.set_xlim([-1.5,1.5])
axes.set_ylim([-1.5,1.5])
plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, np.squeeze(train_Y))

6.结论

 

二、实验2:正则化

1.实验综述

2.导入必要的包

import numpy as np
import matplotlib.pyplot as plt
#reg_utils.py 定义好了一个3层的神经网络结构所包含的必要函数以及本实验中能用到的重要函数
from reg_utils import sigmoid, relu, plot_decision_boundary, initialize_parameters, load_2D_dataset, predict_dec
from reg_utils import compute_cost, predict, forward_propagation, backward_propagation, update_parameters
import sklearn
import sklearn.datasets
import scipy.io
from testCases import * #提供测试用例

%matplotlib inline
#设置绘图的默认风格
plt.rcParams['figure.figsize'] = (7.0, 4.0) 
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

• 可视化待分类数据集

train_X, train_Y, test_X, test_Y = load_2D_dataset()

3.不带正则化的模型

• 神经网络模型

def model(X, Y, learning_rate = 0.3, num_iterations = 30000, print_cost = True, lambd = 0, keep_prob = 1):
    
    '''
    神经网络模型
    
    参数：
    X：训练集样本特征矩阵(n_x,m_train)
    Y:训练集样本标签 0/1 (1,m_train)
    learning_rate:学习率
    num_iterations:梯度下降法迭代次数
    print_cost:为True时，每10000次迭代打印一次cost
    lambd：正则化系数
    keep_prob:每一层单元的保留概率
    
    返回：
    parameters：训练好的参数 字典形式
        parameters["W" + str(l)] = ... 
        parameters["b" + str(l)] = ...
    '''
        
    grads = {}
    costs = []                            # 保存每10000次迭代的cost
    m = X.shape[1]                        # 样本数
    layers_dims = [X.shape[0], 20, 3, 1]  #3层网络 各层的单元数
    
    # 初始化模型参数 返回字典
    parameters = initialize_parameters(layers_dims)

    # 梯度下降法迭代过程

    for i in range(0, num_iterations):

        # 前向传播: LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID.
        if keep_prob == 1: #不使用dropout
            a3, cache = forward_propagation(X, parameters)
        elif keep_prob < 1: #使用dropout
            a3, cache = forward_propagation_with_dropout(X, parameters, keep_prob)
        
        # 代价函数
        if lambd == 0: #不使用L2正则化
            cost = compute_cost(a3, Y)
        else:           #使用L2正则化
            cost = compute_cost_with_regularization(a3, Y, parameters, lambd)
            
        # 反向传播
        assert(lambd==0 or keep_prob==1)    # 可能同时使用L2正则化和dropout
                                            # 本次实验一次只使用一个
        if lambd == 0 and keep_prob == 1: #不使用L2也不使用dropout
            grads = backward_propagation(X, Y, cache)
        elif lambd != 0:  #使用L2正则化
            grads = backward_propagation_with_regularization(X, Y, cache, lambd)
        elif keep_prob < 1:  #使用dropout
            grads = backward_propagation_with_dropout(X, Y, cache, keep_prob)
        
        # 更新参数
        parameters = update_parameters(parameters, grads, learning_rate)
        
        # 每10000次迭代打印/保存一次cost
        if print_cost and i % 10000 == 0:
            print("Cost after iteration {}: {}".format(i, cost))
        if print_cost and i % 1000 == 0:
            costs.append(cost)
    
    # 绘制cost随迭代次数的变化曲线
    plt.plot(costs)
    plt.ylabel('cost')
    plt.xlabel('iterations (x1,000)')
    plt.title("Learning rate =" + str(learning_rate))
    plt.show()
    
    return parameters

• 训练不带正则化的模型

训练一个不带任何正则化的模型，观察它在训练/测试集上的准确率。

parameters = model(train_X, train_Y)
print ("On the training set:")
predictions_train = predict(train_X, train_Y, parameters)
print ("On the test set:")
predictions_test = predict(test_X, test_Y, parameters)

• 绘制决策边界

plt.title("Model without regularization")
axes = plt.gca()
axes.set_xlim([-0.75,0.40])
axes.set_ylim([-0.75,0.65])
plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, np.squeeze(train_Y))

• 总结

4.L2正则化

• 实现带L2正则化的代价函数

#实现带L2正则化的代价函数
def compute_cost_with_regularization(A3, Y, parameters, lambd):
    '''
    实现带L2正则化的代价函数
    
    参数：
    A3:前向传播计算的模型输出(1,m)
    Y:样本真实标签 （1，m）
    parameters:模型参数 字典形式
    lambd：正则化系数
    
    返回：
    cost：计算的带正则化的代价
    '''
    W1 = parameters['W1']
    W2 = parameters['W2']
    W3 = parameters['W3']
    m = A3.shape[1]
    
    cross_entropy_cost = compute_cost(A3,Y)
    
    L2_regularization_cost = 1./m*lambd/2*(np.sum(np.square(W1))+np.sum(np.square(W2))+np.sum(np.square(W3)))
    cost = L2_regularization_cost + cross_entropy_cost
    return cost

• 测试代码

A3, Y_assess, parameters = compute_cost_with_regularization_test_case()

print("cost = " + str(compute_cost_with_regularization(A3, Y_assess, parameters, lambd = 0.1)))

• 实现带L2正则化的反向传播函数

def backward_propagation_with_regularization(X, Y, cache, lambd):
    """
    实现带L2正则化的反向传播
    
    参数：
    X：样本特征矩阵 (nx,m)
    Y: 样本真实标签(1,m)
    cache:前向传播中缓存的中间结果
    lambd:正则化系数
    
    返回：
    grads：每次迭代后模型中参数的梯度 dZ,dW,db
    """
    
    m = X.shape[1]
    (Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3) = cache
    
    dZ3 = A3 - Y
    dW3 = 1./m*np.dot(dZ3,A2.T)+lambd/m*W3
    db3 = 1./m*np.sum(dZ3,axis=1,keepdims=True)
    dA2 = np.dot(W3.T,dZ3)
    
    dZ2 = dA2*np.int64(A2>0)
    dW2 = 1./m*np.dot(dZ2,A1.T)+lambd/m*W2
    db2 = 1./m*np.sum(dZ2,axis=1,keepdims=True)
    dA1 = np.dot(W2.T,dZ2)
    
    dZ1 = dA1*np.int64(A1>0)
    dW1 = 1./m*np.dot(dZ1,X.T)+lambd/m*W1
    db1 = 1./m*np.sum(dZ1,axis=1,keepdims=True)

    
    gradients = {"dZ3": dZ3, "dW3": dW3, "db3": db3,"dA2": dA2,
                 "dZ2": dZ2, "dW2": dW2, "db2": db2, "dA1": dA1, 
                 "dZ1": dZ1, "dW1": dW1, "db1": db1}
    
    return gradients

• 测试代码

X_assess, Y_assess, cache = backward_propagation_with_regularization_test_case()

grads = backward_propagation_with_regularization(X_assess, Y_assess, cache, lambd = 0.7)
print ("dW1 = "+ str(grads["dW1"]))
print ("dW2 = "+ str(grads["dW2"]))
print ("dW3 = "+ str(grads["dW3"]))

• 训练带L2正则化的模型

parameters = model(train_X, train_Y, lambd = 0.7)
print ("On the train set:")
predictions_train = predict(train_X, train_Y, parameters)
print ("On the test set:")
predictions_test = predict(test_X, test_Y, parameters)

模型在测试集上的准确率提升到了93%。而且没有在训练集上出现明显的过拟合。

• 绘制决策边界

plt.title("Model with L2-regularization")
axes = plt.gca()
axes.set_xlim([-0.75,0.40])
axes.set_ylim([-0.75,0.65])
plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, np.squeeze(train_Y))

• 总结

5.Dropout

dropout是在深度学习中广泛使用的正则化技术。 它在每次迭代中使一些神经单元随机失活.

当你使一些关闭一些神经元, 实际上就简化了模型. dropout的思想就是对于每次迭代，训练的是一个不同的模型(只使用神经元的一个子集). 使用dropout，神经元对之前其他神经元的激活输出不再那么敏感(不会赋予很大的权重),因为这些神经元随时都可能关闭

• 带Dropout的前向传播

• 代码实现

def forward_propagation_with_dropout(X, parameters, keep_prob = 0.5):
    """
    实现带dropout的前向传播: LINEAR -> RELU + DROPOUT -> LINEAR -> RELU + DROPOUT -> LINEAR -> SIGMOID.
    
    参数：
    X：输入特征矩阵(nx=2,m)
    参数 -- 字典形式 包含 "W1", "b1", "W2", "b2", "W3", "b3":
                    W1 -- weight matrix of shape (20, 2)
                    b1 -- bias vector of shape (20, 1)
                    W2 -- weight matrix of shape (3, 20)
                    b2 -- bias vector of shape (3, 1)
                    W3 -- weight matrix of shape (1, 3)
                    b3 -- bias vector of shape (1, 1)
    keep_prob - 使用dropout的层的单元保留概率
    
    返回:
    A3 ：模型前向传播输出 (1,m)
    cache :前向传播的缓存结果 便于与反向传播共享参数 元组形式
    """
    
    np.random.seed(1)
    
    W1 = parameters["W1"]
    b1 = parameters["b1"]
    W2 = parameters["W2"]
    b2 = parameters["b2"]
    W3 = parameters["W3"]
    b3 = parameters["b3"]
    
    Z1 = np.dot(W1,X)+b1
    A1 = relu(Z1)
    
    D1 = np.random.rand(A1.shape[0],A1.shape[1])
    D1 = (D1<keep_prob)
    A1 = A1*D1
    A1 /= keep_prob
    
    Z2 = np.dot(W2,A1)+b2
    A2 = relu(Z2)
    
    D2 = np.random.rand(A2.shape[0],A2.shape[1])
    D2 = (D2<keep_prob)
    A2 = A2*D2
    A2 /= keep_prob
    
    Z3 = np.dot(W3,A2)+b3
    A3 = sigmoid(Z3)
    
    
    cache = (Z1, D1, A1, W1, b1, Z2, D2, A2, W2, b2, Z3, A3, W3, b3)
    
    return A3, cache

• 测试代码

X_assess, parameters = forward_propagation_with_dropout_test_case()

A3, cache = forward_propagation_with_dropout(X_assess, parameters, keep_prob = 0.7)
print ("A3 = " + str(A3))

 

• 带Dropout的反向传播

• 代码实现

def backward_propagation_with_dropout(X, Y, cache, keep_prob):
    """
    实现带dropout的反向传播
    
    参数:
    X：输入特征矩阵(nx=2,m)
    Y: 样本真实标签(1,m)
    cache：forward_propagation_with_dropout()缓存的结果，并与共享参数
    keep_prob：使用dropout的层的单元保留概率
    
    返回:
    gradients grads：每次迭代后模型中参数的梯度 dZ,dW,db
    """
    
    m = X.shape[1]
    (Z1, D1, A1, W1, b1, Z2, D2, A2, W2, b2, Z3, A3, W3, b3) = cache
    
    dZ3 = A3 - Y
    dW3 = 1./m*np.dot(dZ3,A2.T)
    db3 = 1./m*np.sum(dZ3,axis=1,keepdims=True)
    dA2 = np.dot(W3.T,dZ3)
    
    dA2 = dA2*D2
    dA2 /= keep_prob
    
    dZ2 = dA2*np.int64(A2>0)
    dW2 = 1./m*np.dot(dZ2,A1.T)
    db2 = 1./m*np.sum(dZ2,axis=1,keepdims=True)
    dA1 = np.dot(W2.T,dZ2)
    
    dA1 = dA1*D1
    dA1 /= keep_prob
    
    dZ1 = dA1*np.int64(A1>0)
    dW1 = 1./m*np.dot(dZ1,X.T)
    db1 = 1./m*np.sum(dZ1,axis=1,keepdims=True)
    
    gradients = {"dZ3": dZ3, "dW3": dW3, "db3": db3,"dA2": dA2,
                 "dZ2": dZ2, "dW2": dW2, "db2": db2, "dA1": dA1, 
                 "dZ1": dZ1, "dW1": dW1, "db1": db1}
    
    return gradients

• 测试代码

X_assess, Y_assess, cache = backward_propagation_with_dropout_test_case()

gradients = backward_propagation_with_dropout(X_assess, Y_assess, cache, keep_prob = 0.8)

print ("dA1 = " + str(gradients["dA1"]))
print ("dA2 = " + str(gradients["dA2"]))

• 训练带有Dropout的mox

parameters = model(train_X, train_Y, keep_prob = 0.86, learning_rate = 0.3)

print ("On the train set:")
predictions_train = predict(train_X, train_Y, parameters)
print ("On the test set:")
predictions_test = predict(test_X, test_Y, parameters)

Dropout效果非常好，模型在测试集上的准确率达到了95%，模型不仅在训练集上不再过拟合，而且在测试集上效果也非常好。

• 绘制决策边界

plt.title("Model with dropout")
axes = plt.gca()
axes.set_xlim([-0.75,0.40])
axes.set_ylim([-0.75,0.65])
plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, np.squeeze(train_Y))

 

• 总结

6.结论

三、实验3:梯度检查

1.实验综述

2.导入必要的包

import numpy as np
from testCases import * #提供测试用例
#gc_utils.py 提供梯度检验实验需要的函数
from gc_utils import sigmoid, relu, dictionary_to_vector, vector_to_dictionary, gradients_to_vector

3.梯度检验时怎么工作的？

• 一维梯度检验

• 代码实现

def forward_propagation(x, theta):
    """
    实现线性前向传播就算 (J(theta) = theta * x)
    
    参数:
    x -- 实数输入
    theta -- 模型参数，也是实数
    
    返回:
    J --  J(theta) = theta * x
    """
    
    J = theta*x
    
    
    return J

def backward_propagation(x, theta):
    """
    计算J对theta的梯度 (see Figure 1).
    
    参数:
    x -- 实数输入
    theta -- 模型参数，也是实数
    
    返回:
    dtheta -- J对theta的梯度
    """
    
    dtheta = x
    
    return dtheta

• 实现梯度检查

• 代码实现

def gradient_check(x, theta, epsilon = 1e-7):
    """
    实现梯度检验
    
    参数:
    x -- 实数输入
    theta -- 模型参数，也是实数
    epsilon -- 非常小的数
    
    返回:
    difference -- 近似梯度和反向传播计算的梯度的距离
    """
    
    thetaplus = theta + epsilon
    thetaminus = theta - epsilon
    J_plus = forward_propagation(x,thetaplus)
    J_minus = forward_propagation(x,thetaminus)
    gradapprox = (J_plus-J_minus)/(2*epsilon)
    
    grad = backward_propagation(x,theta)
    
    numerator = np.linalg.norm(grad - gradapprox)
    denominator = np.linalg.norm(grad)+np.linalg.norm(gradapprox)
    difference = numerator/denominator
    
    if difference < 1e-7:
        print ("The gradient is correct!")
    else:
        print ("The gradient is wrong!")
    
    return difference

 

x, theta = 2, 4
difference = gradient_check(x, theta)
print("difference = " + str(difference))

• N维梯度检验

 

• 代码实现

def forward_propagation_n(X, Y, parameters):
    """
    实现上图中3层神经网络的前向传播
    
    参数：
    X -- 样本特征矩阵 (nx,m)
    Y -- 样本真实标签 (1,m)
    模型参数 -- 字典形式 包含 "W1", "b1", "W2", "b2", "W3", "b3":
                    W1 -- weight matrix of shape (5, 4)
                    b1 -- bias vector of shape (5, 1)
                    W2 -- weight matrix of shape (3, 5)
                    b2 -- bias vector of shape (3, 1)
                    W3 -- weight matrix of shape (1, 3)
                    b3 -- bias vector of shape (1, 1)
    
    返回：
    cost -- m个样本的cost值
    """
    
    m = X.shape[1]
    W1 = parameters["W1"]
    b1 = parameters["b1"]
    W2 = parameters["W2"]
    b2 = parameters["b2"]
    W3 = parameters["W3"]
    b3 = parameters["b3"]

    # LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID
    Z1 = np.dot(W1,X)+b1
    A1 = relu(Z1)
    Z2 = np.dot(W2,A1)+b2
    A2 = relu(Z2)
    Z3 = np.dot(W3,A2)+b3
    A3 = sigmoid(Z3)

    # Cost
    logprobs = -Y*np.log(A3)-(1-Y)*np.log(1-A3)
    cost = 1./m*np.sum(logprobs)
    
    cache = (Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3)
    
    return cost, cache

def backward_propagation_n(X, Y, cache):
 
    """
    实现上图中3层神经网络的反向传播
    
    参数：
    X -- 样本特征矩阵 (nx,m)
    Y -- 样本真实标签 (1,m)
    cache -- 前向传播中缓存的值 参数共享
    
    返回：
    gradients -- 字典形式 模型参数的梯度 dZ,dW,db
    """
    
    m = X.shape[1]
    (Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3) = cache
    
    dZ3 = A3 - Y
    dW3 = 1./m*np.dot(dZ3,A2.T)
    db3 = 1./m*np.sum(dZ3,axis=1,keepdims=True)
    dA2 = np.dot(W3.T,dZ3)
    
    dZ2 = dA2*np.int64(A2>0)
    dW2 = 1./m*np.dot(dZ2,A1.T)
    db2 = 1./m*np.sum(dZ2,axis=1,keepdims=True)
    dA1 = np.dot(W2.T,dZ2)
    
    dZ1 = dA1*np.int64(A1>0)
    dW1 = 1./m*np.dot(dZ1,X.T)
    db1 = 1./m*np.sum(dZ1,axis=1,keepdims=True)
    
    gradients = {"dZ3": dZ3, "dW3": dW3, "db3": db3,
                 "dA2": dA2, "dZ2": dZ2, "dW2": dW2, "db2": db2,
                 "dA1": dA1, "dZ1": dZ1, "dW1": dW1, "db1": db1}
    
    return gradients

• 梯度检查工作原理

• 实现梯度检查

• 代码实现

# GRADED FUNCTION: gradient_check_n

def gradient_check_n(parameters, gradients, X, Y, epsilon = 1e-7):
    """
    实现n维梯度检查
    
    参数：
    parameters -- Python字典 包含模型所有参数 "W1", "b1", "W2", "b2", "W3", "b3":
    grad -- 反向传播计算的参数梯度，字典形式，和参数对应
    X -- 样本特征矩阵 (nx,m)
    Y -- 样本真实标签 (1,m)
    epsilon -- 一个很小的偏移值
    
    返回:
    difference -- 近似梯度向量和反向传播计算的梯度向量的距离
    """
    parameters_values,_ = dictionary_to_vector(parameters) #把模型参数拉伸为向量
    grad = gradients_to_vector(gradients) #把反向传播计算的梯度拉伸为向量
    num_parameters = len(parameters_values)#参数数量
    J_plus = np.zeros((num_parameters,1))
    J_minus = np.zeros((num_parameters,1))
    gradapprox = np.zeros((num_parameters,1))
    
    #计算每个参数的近似梯度
    for i in range(num_parameters):
        theta_plus = np.copy(parameters_values)
        theta_plus[i][0] = theta_plus[i][0]+epsilon
        J_plus[i],_ = forward_propagation_n(X,Y,vector_to_dictionary(theta_plus))
        
        theta_minus = np.copy(parameters_values)
        theta_minus[i][0] = theta_minus[i][0]-epsilon
        J_minus[i],_ = forward_propagation_n(X,Y,vector_to_dictionary(theta_minus))
        
        gradapprox[i] = (J_plus[i]-J_minus[i])/(2*epsilon)
        
    difference = np.linalg.norm(grad-gradapprox)/(np.linalg.norm(grad)+np.linalg.norm(gradapprox))
    

    if difference > 1e-7:
        print ("\033[93m" + "There is a mistake in the backward propagation! difference = " + str(difference) + "\033[0m")
    else:
        print ("\033[92m" + "Your backward propagation works perfectly fine! difference = " + str(difference) + "\033[0m")
    
    return difference

X, Y, parameters = gradient_check_n_test_case()

cost, cache = forward_propagation_n(X, Y, parameters)
gradients = backward_propagation_n(X, Y, cache)
difference = gradient_check_n(parameters, gradients, X, Y)

• 总结

 

四、实验完整代码

实验完整代码       下载密码:pfdo