深度学习 第二课 第一周 作业1

最新推荐文章于 2023-02-15 15:58:25 发布
最新推荐文章于 2023-02-15 15:58:25 发布
阅读量432 收藏
点赞数
分类专栏： 深度学习 Python
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/Anakin6174/article/details/98035231
版权

Initialization

Welcome to the first assignment of “Improving Deep Neural Networks”.

Training your neural network requires specifying an initial value of the weights. A well chosen initialization method will help learning.

If you completed the previous course of this specialization, you probably followed our instructions for weight initialization, and it has worked out so far. But how do you choose the initialization for a new neural network? In this notebook, you will see how different initializations lead to different results.

A well chosen initialization can:

  • Speed up the convergence of gradient descent
  • Increase the odds of gradient descent converging to a lower training (and generalization) error

To get started, run the following cell to load the packages and the planar dataset you will try to classify.

import numpy as np
import matplotlib.pyplot as plt
import sklearn
import sklearn.datasets
from init_utils import sigmoid, relu, compute_loss, forward_propagation, backward_propagation
from init_utils import update_parameters, predict, load_dataset, plot_decision_boundary, predict_dec

%matplotlib inline
plt.rcParams['figure.figsize'] = (7.0, 4.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# load image dataset: blue/red dots in circles
train_X, train_Y, test_X, test_Y = load_dataset()

C:\ProgramData\Anaconda3\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters

[外链图片转存失败(img-R4qUtnwj-1564626083337)(output_1_1.png)]

You would like a classifier to separate the blue dots from the red dots.

1 - Neural Network model

You will use a 3-layer neural network (already implemented for you). Here are the initialization methods you will experiment with:

  • Zeros initialization – setting initialization = "zeros" in the input argument.
  • Random initialization – setting initialization = "random" in the input argument. This initializes the weights to large random values.
  • He initialization – setting initialization = "he" in the input argument. This initializes the weights to random values scaled according to a paper by He et al., 2015.

Instructions: Please quickly read over the code below, and run it. In the next part you will implement the three initialization methods that this model() calls.

def model(X, Y, learning_rate = 0.01, num_iterations = 15000, print_cost = True, initialization = "he"):
    """
    Implements a three-layer neural network: LINEAR->RELU->LINEAR->RELU->LINEAR->SIGMOID.
    
    Arguments:
    X -- input data, of shape (2, number of examples)
    Y -- true "label" vector (containing 0 for red dots; 1 for blue dots), of shape (1, number of examples)
    learning_rate -- learning rate for gradient descent 
    num_iterations -- number of iterations to run gradient descent
    print_cost -- if True, print the cost every 1000 iterations
    initialization -- flag to choose which initialization to use ("zeros","random" or "he")
    
    Returns:
    parameters -- parameters learnt by the model
    """
        
    grads = {}
    costs = [] # to keep track of the loss
    m = X.shape[1] # number of examples
    layers_dims = [X.shape[0], 10, 5, 1]
    
    # Initialize parameters dictionary.
    if initialization == "zeros":
        parameters = initialize_parameters_zeros(layers_dims)
    elif initialization == "random":
        parameters = initialize_parameters_random(layers_dims)
    elif initialization == "he":
        parameters = initialize_parameters_he(layers_dims)

    # Loop (gradient descent)

    for i in range(0, num_iterations):

        # Forward propagation: LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID.
        a3, cache = forward_propagation(X, parameters)
        
        # Loss
        cost = compute_loss(a3, Y)

        # Backward propagation.
        grads = backward_propagation(X, Y, cache)
        
        # Update parameters.
        parameters = update_parameters(parameters, grads, learning_rate)
        
        # Print the loss every 1000 iterations
        if print_cost and i % 1000 == 0:
            print("Cost after iteration {}: {}".format(i, cost))
            costs.append(cost)
            
    # plot the loss
    plt.plot(costs)
    plt.ylabel('cost')
    plt.xlabel('iterations (per hundreds)')
    plt.title("Learning rate =" + str(learning_rate))
    plt.show()
    
    return parameters

2 - Zero initialization

There are two types of parameters to initialize in a neural network:

  • the weight matrices ( W [ 1 ] , W [ 2 ] , W [ 3 ] , . . . , W [ L − 1 ] , W [ L ] ) (W^{[1]}, W^{[2]}, W^{[3]}, ..., W^{[L-1]}, W^{[L]}) (W[1],W[2],W[3],...,W[L−1],W[L])
  • the bias vectors ( b [ 1 ] , b [ 2 ] , b [ 3 ] , . . . , b [ L − 1 ] , b [ L ] ) (b^{[1]}, b^{[2]}, b^{[3]}, ..., b^{[L-1]}, b^{[L]}) (b[1],b[2],b[3],...,b[L−1],b[L])

Exercise: Implement the following function to initialize all parameters to zeros. You’ll see later that this does not work well since it fails to “break symmetry”, but lets try it anyway and see what happens. Use np.zeros((…,…)) with the correct shapes.

# GRADED FUNCTION: initialize_parameters_zeros 

def initialize_parameters_zeros(layers_dims):
    """
    Arguments:
    layer_dims -- python array (list) containing the size of each layer.
    
    Returns:
    parameters -- python dictionary containing your parameters "W1", "b1", ..., "WL", "bL":
                    W1 -- weight matrix of shape (layers_dims[1], layers_dims[0])
                    b1 -- bias vector of shape (layers_dims[1], 1)
                    ...
                    WL -- weight matrix of shape (layers_dims[L], layers_dims[L-1])
                    bL -- bias vector of shape (layers_dims[L], 1)
    """
    
    parameters = {}
    L = len(layers_dims)            # number of layers in the network
    
    for l in range(1, L):
        ### START CODE HERE ### (≈ 2 lines of code)
        parameters['W' + str(l)] = np.zeros((layers_dims[l],layers_dims[l-1]))
        parameters['b' + str(l)] = np.zeros((layers_dims[l],1))
        ### END CODE HERE ###
    return parameters

parameters = initialize_parameters_zeros([3,2,1])
print("W1 = " + str(parameters["W1"]))
print("b1 = " + str(parameters["b1"]))
print("W2 = " + str(parameters["W2"]))
print("b2 = " + str(parameters["b2"]))

W1 = [[0. 0. 0.]
 [0. 0. 0.]]
b1 = [[0.]
 [0.]]
W2 = [[0. 0.]]
b2 = [[0.]]

Expected Output:

**W1** [[ 0. 0. 0.] [ 0. 0. 0.]]
**b1** [[ 0.] [ 0.]]
**W2** [[ 0. 0.]]
**b2** [[ 0.]]

Run the following code to train your model on 15,000 iterations using zeros initialization.

parameters = model(train_X, train_Y, initialization = "zeros")
print ("On the train set:")
predictions_train = predict(train_X, train_Y, parameters)
print ("On the test set:")
predictions_test = predict(test_X, test_Y, parameters)

Cost after iteration 0: 0.6931471805599453
Cost after iteration 1000: 0.6931471805599453
Cost after iteration 2000: 0.6931471805599453
Cost after iteration 3000: 0.6931471805599453
Cost after iteration 4000: 0.6931471805599453
Cost after iteration 5000: 0.6931471805599453
Cost after iteration 6000: 0.6931471805599453
Cost after iteration 7000: 0.6931471805599453
Cost after iteration 8000: 0.6931471805599453
Cost after iteration 9000: 0.6931471805599453
Cost after iteration 10000: 0.6931471805599455
Cost after iteration 11000: 0.6931471805599453
Cost after iteration 12000: 0.6931471805599453
Cost after iteration 13000: 0.6931471805599453
Cost after iteration 14000: 0.6931471805599453

[外链图片转存失败(img-oXGo3WAI-1564626083341)(output_11_1.png)]

On the train set:
Accuracy: 0.5
On the test set:
Accuracy: 0.5

The performance is really bad, and the cost does not really decrease, and the algorithm performs no better than random guessing. Why? Lets look at the details of the predictions and the decision boundary:

print ("predictions_train = " + str(predictions_train))
print ("predictions_test = " + str(predictions_test))

predictions_train = [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  0 0 0 0 0 0 0 0 0 0 0 0]]
predictions_test = [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]]

plt.title("Model with Zeros initialization")
axes = plt.gca()
axes.set_xlim([-1.5,1.5])
axes.set_ylim([-1.5,1.5])
plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, train_Y)

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\colors.py in to_rgba(c, alpha)
    165     try:
--> 166         rgba = _colors_full_map.cache[c, alpha]
    167     except (KeyError, TypeError):  # Not in cache, or unhashable.


TypeError: unhashable type: 'numpy.ndarray'


During handling of the above exception, another exception occurred:


ValueError                                Traceback (most recent call last)

C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\axes\_axes.py in scatter(self, x, y, s, c, marker, cmap, norm, vmin, vmax, alpha, linewidths, verts, edgecolors, **kwargs)
   4273                 # must be acceptable as PathCollection facecolors
-> 4274                 colors = mcolors.to_rgba_array(c)
   4275             except ValueError:


C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\colors.py in to_rgba_array(c, alpha)
    266     for i, cc in enumerate(c):
--> 267         result[i] = to_rgba(cc, alpha)
    268     return result


C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\colors.py in to_rgba(c, alpha)
    167     except (KeyError, TypeError):  # Not in cache, or unhashable.
--> 168         rgba = _to_rgba_no_colorcycle(c, alpha)
    169         try:


C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\colors.py in _to_rgba_no_colorcycle(c, alpha)
    222     if len(c) not in [3, 4]:
--> 223         raise ValueError("RGBA sequence should have length 3 or 4")
    224     if len(c) == 3 and alpha is None:


ValueError: RGBA sequence should have length 3 or 4


During handling of the above exception, another exception occurred:


ValueError                                Traceback (most recent call last)

<ipython-input-7-0ec6f7c907fb> in <module>()
      3 axes.set_xlim([-1.5,1.5])
      4 axes.set_ylim([-1.5,1.5])
----> 5 plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, train_Y)


E:\CodePratice\Jupyter\第二课第一周编程作业\assignment1\init_utils.py in plot_decision_boundary(model, X, y)
    215     plt.ylabel('x2')
    216     plt.xlabel('x1')
--> 217     plt.scatter(X[0, :], X[1, :], c=y, cmap=plt.cm.Spectral)
    218     plt.show()
    219 


C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\pyplot.py in scatter(x, y, s, c, marker, cmap, norm, vmin, vmax, alpha, linewidths, verts, edgecolors, hold, data, **kwargs)
   3468                          vmin=vmin, vmax=vmax, alpha=alpha,
   3469                          linewidths=linewidths, verts=verts,
-> 3470                          edgecolors=edgecolors, data=data, **kwargs)
   3471     finally:
   3472         ax._hold = washold


C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\__init__.py in inner(ax, *args, **kwargs)
   1853                         "the Matplotlib list!)" % (label_namer, func.__name__),
   1854                         RuntimeWarning, stacklevel=2)
-> 1855             return func(ax, *args, **kwargs)
   1856 
   1857         inner.__doc__ = _add_data_doc(inner.__doc__,


C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\axes\_axes.py in scatter(self, x, y, s, c, marker, cmap, norm, vmin, vmax, alpha, linewidths, verts, edgecolors, **kwargs)
   4277                 raise ValueError("c of shape {} not acceptable as a color "
   4278                                  "sequence for x with size {}, y with size {}"
-> 4279                                  .format(c.shape, x.size, y.size))
   4280         else:
   4281             colors = None  # use cmap, norm after collection is created


ValueError: c of shape (1, 300) not acceptable as a color sequence for x with size 300, y with size 300

[外链图片转存失败(img-dZ2g0l1F-1564626083342)(output_14_1.png)]

The model is predicting 0 for every example.

In general, initializing all the weights to zero results in the network failing to break symmetry. This means that every neuron in each layer will learn the same thing, and you might as well be training a neural network with n [ l ] = 1 n^{[l]}=1 n[l]=1 for every layer, and the network is no more powerful than a linear classifier such as logistic regression.

**What you should remember**: - The weights $W^{[l]}$ should be initialized randomly to break symmetry. - It is however okay to initialize the biases $b^{[l]}$ to zeros. Symmetry is still broken so long as $W^{[l]}$ is initialized randomly.

3 - Random initialization

To break symmetry, lets intialize the weights randomly. Following random initialization, each neuron can then proceed to learn a different function of its inputs. In this exercise, you will see what happens if the weights are intialized randomly, but to very large values.

Exercise: Implement the following function to initialize your weights to large random values (scaled by *10) and your biases to zeros. Use np.random.randn(..,..) * 10 for weights and np.zeros((.., ..)) for biases. We are using a fixed np.random.seed(..) to make sure your “random” weights match ours, so don’t worry if running several times your code gives you always the same initial values for the parameters.

# GRADED FUNCTION: initialize_parameters_random

def initialize_parameters_random(layers_dims):
    """
    Arguments:
    layer_dims -- python array (list) containing the size of each layer.
    
    Returns:
    parameters -- python dictionary containing your parameters "W1", "b1", ..., "WL", "bL":
                    W1 -- weight matrix of shape (layers_dims[1], layers_dims[0])
                    b1 -- bias vector of shape (layers_dims[1], 1)
                    ...
                    WL -- weight matrix of shape (layers_dims[L], layers_dims[L-1])
                    bL -- bias vector of shape (layers_dims[L], 1)
    """
    
    np.random.seed(3)               # This seed makes sure your "random" numbers will be the as ours
    parameters = {}
    L = len(layers_dims)            # integer representing the number of layers
    
    for l in range(1, L):
        ### START CODE HERE ### (≈ 2 lines of code)
        parameters['W' + str(l)] = np.random.rand(layers_dims[l],layers_dims[l-1])*10
        parameters['b' + str(l)] = np.zeros((layers_dims[l],1))
        ### END CODE HERE ###

    return parameters

parameters = initialize_parameters_random([3, 2, 1])
print("W1 = " + str(parameters["W1"]))
print("b1 = " + str(parameters["b1"]))
print("W2 = " + str(parameters["W2"]))
print("b2 = " + str(parameters["b2"]))

W1 = [[5.50797903 7.08147823 2.90904739]
 [5.10827605 8.92946954 8.96293089]]
b1 = [[0.]
 [0.]]
W2 = [[1.2558531  2.07242878]]
b2 = [[0.]]

Expected Output:

**W1** [[ 17.88628473 4.36509851 0.96497468] [-18.63492703 -2.77388203 -3.54758979]]
**b1** [[ 0.] [ 0.]]
**W2** [[-0.82741481 -6.27000677]]
**b2** [[ 0.]]

Run the following code to train your model on 15,000 iterations using random initialization.

parameters = model(train_X, train_Y, initialization = "random")
print ("On the train set:")
predictions_train = predict(train_X, train_Y, parameters)
print ("On the test set:")
predictions_test = predict(test_X, test_Y, parameters)

E:\CodePratice\Jupyter\第二课第一周编程作业\assignment1\init_utils.py:145: RuntimeWarning: divide by zero encountered in log
  logprobs = np.multiply(-np.log(a3),Y) + np.multiply(-np.log(1 - a3), 1 - Y)
E:\CodePratice\Jupyter\第二课第一周编程作业\assignment1\init_utils.py:145: RuntimeWarning: invalid value encountered in multiply
  logprobs = np.multiply(-np.log(a3),Y) + np.multiply(-np.log(1 - a3), 1 - Y)


Cost after iteration 0: inf
Cost after iteration 1000: 0.7639258316786469
Cost after iteration 2000: 0.6693421032050622
Cost after iteration 3000: 0.6514283780249253
Cost after iteration 4000: 0.6470375349403844
Cost after iteration 5000: 0.6447468658343389
Cost after iteration 6000: 0.6411180263961129
Cost after iteration 7000: 0.6283315723088351
Cost after iteration 8000: 0.5970596799052974
Cost after iteration 9000: 0.5503881016247947
Cost after iteration 10000: 0.5074075430668982
Cost after iteration 11000: 0.4903043314405396
Cost after iteration 12000: 0.4790370300361686
Cost after iteration 13000: 0.4744742623800744
Cost after iteration 14000: 0.47142853850552513

[外链图片转存失败(img-TPO8c4Iz-1564626083343)(output_22_2.png)]

On the train set:
Accuracy: 0.6833333333333333
On the test set:
Accuracy: 0.67

If you see “inf” as the cost after the iteration 0, this is because of numerical roundoff; a more numerically sophisticated implementation would fix this. But this isn’t worth worrying about for our purposes.

Anyway, it looks like you have broken symmetry, and this gives better results. than before. The model is no longer outputting all 0s.

print (predictions_train)
print (predictions_test)

[[1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0 1 0 1 1 1 1 0 1 1 1 1 1
  1 1 1 1 1 1 1 0 1 1 1 1 0 1 0 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 0 1 1 1 1 0
  1 1 0 1 1 0 1 0 1 1 1 1 0 1 1 1 1 1 1 0 0 1 1 1 0 1 1 1 1 0 1 1 1 1 1 1
  1 0 1 1 1 0 1 1 0 1 1 1 1 1 1 0 1 0 0 1 1 1 1 1 1 1 1 1 0 1 1 0 0 1 1 1
  1 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 0 1 1 1 1 1 1 1 1
  1 1 1 0 1 0 1 1 1 1 1 1 1 0 1 1 0 1 1 0 1 0 1 1 1 1 1 1 1 0 1 0 1 1 0 1
  1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 1 1 1 1 1 1 0 1 1 1 0 1 1
  1 1 1 1 1 0 1 1 0 1 0 1 0 0 1 1 0 1 1 1 1 1 1 0 0 0 0 1 1 1 1 1 1 1 1 1
  1 1 1 1 1 0 1 1 1 1 1 0]]
[[1 1 1 1 1 1 0 1 1 0 1 1 1 0 0 1 1 1 1 1 1 0 1 1 1 0 1 1 1 1 1 0 1 0 1 1
  0 1 1 0 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1
  1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 0 1 0 1 1 1 1 1 0 1]]

plt.title("Model with large random initialization")
axes = plt.gca()
axes.set_xlim([-1.5,1.5])
axes.set_ylim([-1.5,1.5])
plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, train_Y)

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\colors.py in to_rgba(c, alpha)
    165     try:
--> 166         rgba = _colors_full_map.cache[c, alpha]
    167     except (KeyError, TypeError):  # Not in cache, or unhashable.


TypeError: unhashable type: 'numpy.ndarray'


During handling of the above exception, another exception occurred:


ValueError                                Traceback (most recent call last)

C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\axes\_axes.py in scatter(self, x, y, s, c, marker, cmap, norm, vmin, vmax, alpha, linewidths, verts, edgecolors, **kwargs)
   4273                 # must be acceptable as PathCollection facecolors
-> 4274                 colors = mcolors.to_rgba_array(c)
   4275             except ValueError:


C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\colors.py in to_rgba_array(c, alpha)
    266     for i, cc in enumerate(c):
--> 267         result[i] = to_rgba(cc, alpha)
    268     return result


C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\colors.py in to_rgba(c, alpha)
    167     except (KeyError, TypeError):  # Not in cache, or unhashable.
--> 168         rgba = _to_rgba_no_colorcycle(c, alpha)
    169         try:


C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\colors.py in _to_rgba_no_colorcycle(c, alpha)
    222     if len(c) not in [3, 4]:
--> 223         raise ValueError("RGBA sequence should have length 3 or 4")
    224     if len(c) == 3 and alpha is None:


ValueError: RGBA sequence should have length 3 or 4


During handling of the above exception, another exception occurred:


ValueError                                Traceback (most recent call last)

<ipython-input-13-7e9ab4692133> in <module>()
      3 axes.set_xlim([-1.5,1.5])
      4 axes.set_ylim([-1.5,1.5])
----> 5 plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, train_Y)


E:\CodePratice\Jupyter\第二课第一周编程作业\assignment1\init_utils.py in plot_decision_boundary(model, X, y)
    215     plt.ylabel('x2')
    216     plt.xlabel('x1')
--> 217     plt.scatter(X[0, :], X[1, :], c=y, cmap=plt.cm.Spectral)
    218     plt.show()
    219 


C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\pyplot.py in scatter(x, y, s, c, marker, cmap, norm, vmin, vmax, alpha, linewidths, verts, edgecolors, hold, data, **kwargs)
   3468                          vmin=vmin, vmax=vmax, alpha=alpha,
   3469                          linewidths=linewidths, verts=verts,
-> 3470                          edgecolors=edgecolors, data=data, **kwargs)
   3471     finally:
   3472         ax._hold = washold


C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\__init__.py in inner(ax, *args, **kwargs)
   1853                         "the Matplotlib list!)" % (label_namer, func.__name__),
   1854                         RuntimeWarning, stacklevel=2)
-> 1855             return func(ax, *args, **kwargs)
   1856 
   1857         inner.__doc__ = _add_data_doc(inner.__doc__,


C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\axes\_axes.py in scatter(self, x, y, s, c, marker, cmap, norm, vmin, vmax, alpha, linewidths, verts, edgecolors, **kwargs)
   4277                 raise ValueError("c of shape {} not acceptable as a color "
   4278                                  "sequence for x with size {}, y with size {}"
-> 4279                                  .format(c.shape, x.size, y.size))
   4280         else:
   4281             colors = None  # use cmap, norm after collection is created


ValueError: c of shape (1, 300) not acceptable as a color sequence for x with size 300, y with size 300

[外链图片转存失败(img-Q30fkTMZ-1564626083344)(output_25_1.png)]

Observations:

  • The cost starts very high. This is because with large random-valued weights, the last activation (sigmoid) outputs results that are very close to 0 or 1 for some examples, and when it gets that example wrong it incurs a very high loss for that example. Indeed, when log ⁡ ( a [ 3 ] ) = log ⁡ ( 0 ) \log(a^{[3]}) = \log(0) log(a[3])=log(0), the loss goes to infinity.
  • Poor initialization can lead to vanishing/exploding gradients, which also slows down the optimization algorithm.
  • If you train this network longer you will see better results, but initializing with overly large random numbers slows down the optimization.
**In summary**: - Initializing weights to very large random values does not work well. - Hopefully intializing with small random values does better. The important question is: how small should be these random values be? Lets find out in the next part!

4 - He initialization

Finally, try “He Initialization”; this is named for the first author of He et al., 2015. (If you have heard of “Xavier initialization”, this is similar except Xavier initialization uses a scaling factor for the weights W [ l ] W^{[l]} W[l] of sqrt(1./layers_dims[l-1]) where He initialization would use sqrt(2./layers_dims[l-1]).)

Exercise: Implement the following function to initialize your parameters with He initialization.

Hint: This function is similar to the previous initialize_parameters_random(...). The only difference is that instead of multiplying np.random.randn(..,..) by 10, you will multiply it by 2 dimension of the previous layer \sqrt{\frac{2}{\text{dimension of the previous layer}}} dimension of the previous layer2​ ​, which is what He initialization recommends for layers with a ReLU activation.

# GRADED FUNCTION: initialize_parameters_he

def initialize_parameters_he(layers_dims):
    """
    Arguments:
    layer_dims -- python array (list) containing the size of each layer.
    
    Returns:
    parameters -- python dictionary containing your parameters "W1", "b1", ..., "WL", "bL":
                    W1 -- weight matrix of shape (layers_dims[1], layers_dims[0])
                    b1 -- bias vector of shape (layers_dims[1], 1)
                    ...
                    WL -- weight matrix of shape (layers_dims[L], layers_dims[L-1])
                    bL -- bias vector of shape (layers_dims[L], 1)
    """
    
    np.random.seed(3)
    parameters = {}
    L = len(layers_dims) - 1 # integer representing the number of layers
     
    for l in range(1, L + 1):
        ### START CODE HERE ### (≈ 2 lines of code)
        parameters['W' + str(l)] = np.random.randn(layers_dims[l],layers_dims[l-1])*np.sqrt(2/layers_dims[l-1])
        parameters['b' + str(l)] = np.zeros((layers_dims[l],1))
        ### END CODE HERE ###
        
    return parameters

parameters = initialize_parameters_he([2, 4, 1])
print("W1 = " + str(parameters["W1"]))
print("b1 = " + str(parameters["b1"]))
print("W2 = " + str(parameters["W2"]))
print("b2 = " + str(parameters["b2"]))

W1 = [[ 1.78862847  0.43650985]
 [ 0.09649747 -1.8634927 ]
 [-0.2773882  -0.35475898]
 [-0.08274148 -0.62700068]]
b1 = [[0.]
 [0.]
 [0.]
 [0.]]
W2 = [[-0.03098412 -0.33744411 -0.92904268  0.62552248]]
b2 = [[0.]]

Expected Output:

**W1** [[ 1.78862847 0.43650985] [ 0.09649747 -1.8634927 ] [-0.2773882 -0.35475898] [-0.08274148 -0.62700068]]
**b1** [[ 0.] [ 0.] [ 0.] [ 0.]]
**W2** [[-0.03098412 -0.33744411 -0.92904268 0.62552248]]
**b2** [[ 0.]]

Run the following code to train your model on 15,000 iterations using He initialization.

parameters = model(train_X, train_Y, initialization = "he")
print ("On the train set:")
predictions_train = predict(train_X, train_Y, parameters)
print ("On the test set:")
predictions_test = predict(test_X, test_Y, parameters)

Cost after iteration 0: 0.8830537463419761
Cost after iteration 1000: 0.6879825919728063
Cost after iteration 2000: 0.6751286264523371
Cost after iteration 3000: 0.6526117768893807
Cost after iteration 4000: 0.6082958970572937
Cost after iteration 5000: 0.5304944491717495
Cost after iteration 6000: 0.4138645817071793
Cost after iteration 7000: 0.3117803464844441
Cost after iteration 8000: 0.23696215330322556
Cost after iteration 9000: 0.18597287209206828
Cost after iteration 10000: 0.15015556280371808
Cost after iteration 11000: 0.12325079292273548
Cost after iteration 12000: 0.09917746546525937
Cost after iteration 13000: 0.08457055954024274
Cost after iteration 14000: 0.07357895962677366

[外链图片转存失败(img-XuLznXpc-1564626083345)(output_32_1.png)]

On the train set:
Accuracy: 0.9933333333333333
On the test set:
Accuracy: 0.96

plt.title("Model with He initialization")
axes = plt.gca()
axes.set_xlim([-1.5,1.5])
axes.set_ylim([-1.5,1.5])
plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, train_Y)

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\colors.py in to_rgba(c, alpha)
    165     try:
--> 166         rgba = _colors_full_map.cache[c, alpha]
    167     except (KeyError, TypeError):  # Not in cache, or unhashable.


TypeError: unhashable type: 'numpy.ndarray'


During handling of the above exception, another exception occurred:


ValueError                                Traceback (most recent call last)

C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\axes\_axes.py in scatter(self, x, y, s, c, marker, cmap, norm, vmin, vmax, alpha, linewidths, verts, edgecolors, **kwargs)
   4273                 # must be acceptable as PathCollection facecolors
-> 4274                 colors = mcolors.to_rgba_array(c)
   4275             except ValueError:


C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\colors.py in to_rgba_array(c, alpha)
    266     for i, cc in enumerate(c):
--> 267         result[i] = to_rgba(cc, alpha)
    268     return result


C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\colors.py in to_rgba(c, alpha)
    167     except (KeyError, TypeError):  # Not in cache, or unhashable.
--> 168         rgba = _to_rgba_no_colorcycle(c, alpha)
    169         try:


C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\colors.py in _to_rgba_no_colorcycle(c, alpha)
    222     if len(c) not in [3, 4]:
--> 223         raise ValueError("RGBA sequence should have length 3 or 4")
    224     if len(c) == 3 and alpha is None:


ValueError: RGBA sequence should have length 3 or 4


During handling of the above exception, another exception occurred:


ValueError                                Traceback (most recent call last)

<ipython-input-19-b75eab11dd68> in <module>()
      3 axes.set_xlim([-1.5,1.5])
      4 axes.set_ylim([-1.5,1.5])
----> 5 plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, train_Y)


E:\CodePratice\Jupyter\第二课第一周编程作业\assignment1\init_utils.py in plot_decision_boundary(model, X, y)
    215     plt.ylabel('x2')
    216     plt.xlabel('x1')
--> 217     plt.scatter(X[0, :], X[1, :], c=y, cmap=plt.cm.Spectral)
    218     plt.show()
    219 


C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\pyplot.py in scatter(x, y, s, c, marker, cmap, norm, vmin, vmax, alpha, linewidths, verts, edgecolors, hold, data, **kwargs)
   3468                          vmin=vmin, vmax=vmax, alpha=alpha,
   3469                          linewidths=linewidths, verts=verts,
-> 3470                          edgecolors=edgecolors, data=data, **kwargs)
   3471     finally:
   3472         ax._hold = washold


C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\__init__.py in inner(ax, *args, **kwargs)
   1853                         "the Matplotlib list!)" % (label_namer, func.__name__),
   1854                         RuntimeWarning, stacklevel=2)
-> 1855             return func(ax, *args, **kwargs)
   1856 
   1857         inner.__doc__ = _add_data_doc(inner.__doc__,


C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\axes\_axes.py in scatter(self, x, y, s, c, marker, cmap, norm, vmin, vmax, alpha, linewidths, verts, edgecolors, **kwargs)
   4277                 raise ValueError("c of shape {} not acceptable as a color "
   4278                                  "sequence for x with size {}, y with size {}"
-> 4279                                  .format(c.shape, x.size, y.size))
   4280         else:
   4281             colors = None  # use cmap, norm after collection is created


ValueError: c of shape (1, 300) not acceptable as a color sequence for x with size 300, y with size 300

[外链图片转存失败(img-FDPmTZtE-1564626083346)(output_33_1.png)]

Observations:

  • The model with He initialization separates the blue and the red dots very well in a small number of iterations.

5 - Conclusions

You have seen three different types of initializations. For the same number of iterations and same hyperparameters the comparison is:

</tr>
    <td>
    3-layer NN with zeros initialization
    </td>
    <td>
    50%
    </td>
    <td>
    fails to break symmetry
    </td>
<tr>
    <td>
    3-layer NN with large random initialization
    </td>
    <td>
    83%
    </td>
    <td>
    too large weights 
    </td>
</tr>
<tr>
    <td>
    3-layer NN with He initialization
    </td>
    <td>
    99%
    </td>
    <td>
    recommended method
    </td>
</tr>
**Model** **Train accuracy** **Problem/Comment**
**What you should remember from this notebook**: - Different initializations lead to different results - Random initialization is used to break symmetry and make sure different hidden units can learn different things - Don't intialize to values that are too large - He initialization works well for networks with ReLU activations.
Anakin6174
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
专栏目录
吴恩达深度学习第二课第一周作业代码
08-02
吴恩达深度学习第二课第一周作业，仅含代码。笔记见博客！
deeplearning.ai-lecture2-week1-Initialization-homework
12-12 487
Initialization Welcome to the first assignment of "Improving Deep Neural Networks". Training your neural network requires specifying an initial value of the weights. A well chosen initialization met
深度学习 第二课 第一周 作业2
Anakin6174的博客
08-01 634
Regularization Welcome to the second assignment of this week. Deep Learning models have so much flexibility and capacity that overfitting can be a serious problem, if the training dataset is not big e...
吴恩达老师的深度学习课程第二课第一周作业
01-15
吴恩达老师在course上开设了一门深度学习的课程的作业 我跟着课程完成了前几次的作业，但是由于不是会员所以没法判分，上传给大家希望大家指正。 这里是第2课第1周的作业。
吴恩达深度学习第二课第一周作业及学习心得体会——初始化权重、L2正则化、dropout
weixin_42604446的博客
08-02 1851
概述 学习课程后，再第一周第四课作业的基础上完成该周作业，现将心得体会记录如下。 初始化权重 代码更改 在之前的建立神经网络的过程中，提到权重w不能为0，而将它初始化为一个随机的值。然而在一个深层神经网络中，当w的值被初始化过大时，进入深层时呈指数型增长，造成梯度爆炸；过小时又会呈指数级衰减，造成梯度消失。 Python中将w进行随机初始化时，使用numpy库中的np.random.ra...
吴恩达深度学习第一课第四周作业及学习心得体会
weixin_42604446的博客
07-27 2406
概述 在完成吴恩达第一课第四周，即L层neuralnetwork学习后，了解了深层神经网络的结构，然后参考网上的资料，完成该周作业，现将心得体会记录如下。 核心代码 调用库如下所示 import numpy as np import matplotlib.pyplot as plt 初始化参数函数 def init_para(layer_dims): L = len(lay...
吴恩达深度学习第一章第二周编程作业
only my railgun
10-31 3326
文章目录前言一、题目描述。二、相关库三、编程步骤1.数据预处理2.模型的封装3.模型的调用4.结果展示总结 前言   本人处于初学阶段，编程能力有限，代码的编写参考了网上的大神。 一、题目描述。   我们需要训练得到一个逻辑回归分类器来对图片进行二分类（是猫和不是猫），通过正向传播和反向传播来对参数进行优化。官方网站给出的题目形式是填空编程，这里我将给出完整的代码，可能和官网上的有些出入，但是大致相同。 二、相关库 1.numpy:强大的科学计算库，主要用于矩阵的计算 2.matplotlib
吴恩达深度学习课程第二章第一周编程作业
only my railgun
11-08 1564
文章目录声明一、任务描述二、编程实现1.数据2.参数初始化2.1 初始化参数为02.2 参数随机初始化2.3 抑梯度异常初始化2.4 主控函数2.5 测试结果对比2.5.1 初始化为02.5.2 随机初始化参数2.5.3抑梯度异常初始化3.模型正则化3.1 L2正则化二、使用步骤1.引入库2.读入数据总结 声明   本博客只是记录一下本人在深度学习过程中的学习笔记和编程经验，大部分代码是参考了【中文】【吴恩达课后编程作业】Course 2 - 改善深层神经网络 - 第一周作业(1&2&3)这
吴恩达深度学习第二课第一周知识总结
yeeanna的博客
06-16 166
吴恩达深度学习第二课知识总结（一） 仅供自己记录整理 1.1 训练，验证，测试 1.2 偏差，方差 偏差：欠拟合 训练集错误率50%，验证集错误率50% 方差：过拟合 训练集错误率1%，验证集错误率50% 1.3 机器学习基础 训练神经网络的方法： 1.4 正则化——L2正则化 L2正则化： 其中lamata是正则化参数，通过调整lamata避免过拟合。 神经网络中的正则化： 如何使用该范数实现梯度下降： 此时的L2范数相当于权重衰减。 1.5 为什么正则化可以防止过拟合？ 不用知道原因，想知道去看
深度学习吴恩达第二门课-改善深层神经网络-第一周作业
ThreeS_tones的博客
02-15 202
深度学习吴恩达第二门课-改善深层神经网络-第一周作业
吴恩达深度学习第二课第一周编程作业
04-11
吴恩达深度学习第二课第一周编程作业，包括数据和代码，可以跑通
吴恩达深度学习第五课第一周第二周及第三周作业作业
11-20
吴恩达深度学习第五课第一周第二周及第三周作业作业，千幸万苦终于找到，每周作业包括选择题及编程题，并且都提供了答案，编程题有两个版本，空白版本和完成版本，大家可以选择使用，所有编程题都亲自运行，没任何...
吴恩达老师的深度学习课程第一课第四周作业
01-15
附件里是如题的作业 我是使用Spyder把作业加载进去，然后新建一个程序，一段一段运行的。用于检验的代码在运行之后一般就注释掉或者删掉了。 后面的代码规范性正在加强，大家看着也方便很多
吴恩达老师的深度学习课程第一课第二周作业
12-28
附件里是如题的作业 我是使用Spyder把作业加载进去，然后新建一个程序，一段一段运行的。用于检验的代码在运行之后一般就注释掉或者删掉了。 当时没有考虑怎么代码的规范性，以后会注意。
吴恩达深度学习第四课第一周 卷积神经网络
qq_44428997的博客
07-17 856
吴恩达深度学习第四课第一周
吴恩达深度学习第一课第三周编程作业（上）
qq_45502052的博客
03-15 1939
作业描述： 用1层隐藏层的神经网络分类二维数据 欢迎来到第3周的编程作业。 现在是时候建立你的第一个神经网络了，它将具有一层隐藏层。 你将看到此模型与你使用逻辑回归实现的模型之间的巨大差异。 你将学到如何： 实现具有单个隐藏层的2分类神经网络 使用具有非线性激活函数的神经元，例如tanh 计算交叉熵损失 实现前向和后向传播 数据集及源码下载： 链接:https://pan.baidu.com/s/1UT6XLCkohr3n37wTwkI3BA 提取码:4bmw 代码中的一些常见函数学习：.
pycharm 显示当前 python 文件下的函数和类的列表
热门推荐
Anakin6174的博客
06-09 1万+
显示每个py文件里面的类和方法，方便快速跳转。 方法一（推荐）： 左侧 project 工具栏窗口顶部那个齿轮有个 show member 选项，默认是不开的，勾选后 py 文件会显示内部定义的 class 和 method 每个文件可以自由选择折叠还是展开 方法二： 只能对选择某个文件来展开 ...
python同级目录导入问题
Anakin6174的博客
11-28 3984
在PyCharm2017中同目录下import其他模块，会出现No model named …的报错 方法： 1，这是因为PyCharm不会将当前文件目录自动加入source_path。 在当前目录右键make_directory as–&gt;Sources Root 2，如果导入的目录没有__init__.py文件，pycharm并不能识别其为Python包，也会出现导入出错，这样只需添加一个...
Python实现switch方法
Anakin6174的博客
03-28 3643
在很多编程语言中都有switch/case语法，但是Python中并没有switch语句。相应的应对方法除了if-else语句外，还可以用字典dict来实现； 举例如下： import outer_fun class PythonSwitch(): def __init__(self): self._operator_map = { 'A': s...
吴恩达《深度学习》l1w2作业2数据集下载
最新发布
07-27
吴恩达教授的《深度学习》课程l1w2作业2需要用到一个数据集进行练习。要下载这个数据集，首先要找到相关的下载链接或来源。 可以通过以下两种方式获取《深度学习》课程的所有数据集。 第一种方式是前往吴恩达教授...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值