吴恩达深度学习课后习题第三课第一周编程作业3:Gradient Checking

Table of Contents

• 1 - Packages
• 2 - Problem Statement
• 3 - How does Gradient Checking work?
• 4 - 1-Dimensional Gradient Checking
  • Exercise 1 - forward_propagation
  • Exercise 2 - backward_propagation
  • Exercise 3 - gradient_check
• 5 - N-Dimensional Gradient Checking
  • Exercise 4 - gradient_check_n

1 - Packages

In [1]:

import numpy as np
from testCases import *
from public_tests import *
from gc_utils import sigmoid, relu, dictionary_to_vector, vector_to_dictionary, gradients_to_vector

%load_ext autoreload
%autoreload 2

2 - Problem Statement

You are part of a team working to make mobile payments available globally, and are asked to build a deep learning model to detect fraud--whenever someone makes a payment, you want to see if the payment might be fraudulent, such as if the user's account has been taken over by a hacker.

You already know that backpropagation is quite challenging to implement, and sometimes has bugs. Because this is a mission-critical application, your company's CEO wants to be really certain that your implementation of backpropagation is correct. Your CEO says, "Give me proof that your backpropagation is actually working!" To give this reassurance, you are going to use "gradient checking."

Let's do it!

3 - How does Gradient Checking work?

Backpropagation computes the gradients ∂J∂θ∂J∂θ, where θθ denotes the parameters of the model. JJ is computed using forward propagation and your loss function.

Because forward propagation is relatively easy to implement, you're confident you got that right, and so you're almost 100% sure that you're computing the cost JJ correctly. Thus, you can use your code for computing JJ to verify the code for computing ∂J∂θ∂J∂θ.

Let's look back at the definition of a derivative (or gradient):

∂J∂θ=limε→0J(θ+ε)−J(θ−ε)2ε(1)(1)∂J∂θ=limε→0J(θ+ε)−J(θ−ε)2ε

If you're not familiar with the "limε→0limε→0" notation, it's just a way of saying "when εε is really, really small."

You know the following:

∂J∂θ∂J∂θ is what you want to make sure you're computing correctly. You can compute J(θ+ε)J(θ+ε) and J(θ−ε)J(θ−ε) (in the case that θθ is a real number), since you're confident your implementation for JJ is correct. Let's use equation (1) and a small value for εε to convince your CEO that your code for computing ∂J∂θ∂J∂θ is correct!

4 - 1-Dimensional Gradient Checking

Consider a 1D linear function J(θ)=θxJ(θ)=θx. The model contains only a single real-valued parameter θθ, and takes xx as input.

You will implement code to compute J(.)J(.) and its derivative ∂J∂θ∂J∂θ. You will then use gradient checking to make sure your derivative computation for JJ is correct.

Figure 1:1D linear model

The diagram above shows the key computation steps: First start with xx, then evaluate the function J(x)J(x) ("forward propagation"). Then compute the derivative ∂J∂θ∂J∂θ ("backward propagation").

Exercise 1 - forward_propagation

Implement forward propagation. For this simple function compute J(.)J(.)

In [2]:

# GRADED FUNCTION: forward_propagation

def forward_propagation(x, theta):
    """
    Implement the linear forward propagation (compute J) presented in Figure 1 (J(theta) = theta * x)
    
    Arguments:
    x -- a real-valued input
    theta -- our parameter, a real number as well
    
    Returns:
    J -- the value of function J, computed using the formula J(theta) = theta * x
    """
    
    # (approx. 1 line)
    # J = 
    # YOUR CODE STARTS HERE
    
    
    J = theta * x
    # YOUR CODE ENDS HERE
    
    return J

In [3]:

x, theta = 2, 4
J =