Sequence Models Week 1 Building a recurrent neural network - step by step

最新推荐文章于 2022-04-03 18:33:33 发布
最新推荐文章于 2022-04-03 18:33:33 发布
阅读量145 收藏
点赞数
文章标签： 人工智能 python
原文链接：http://www.cnblogs.com/RB26DETT/p/11444943.html
版权

Building your Recurrent Neural Network - Step by Step

Welcome to Course 5's first assignment! In this assignment, you will implement your first Recurrent Neural Network in numpy.

Recurrent Neural Networks (RNN) are very effective for Natural Language Processing and other sequence tasks because they have "memory". They can read inputs x⟨t⟩ (such as words) one at a time, and remember some information/context through the hidden layer activations that get passed from one time-step to the next. This allows a uni-directional RNN to take information from the past to process later inputs. A bidirection RNN can take context from both the past and the future.

Notation:

  • Superscript [l] denotes an object associated with the lth layer.

    • Example: a[4] is the 4th layer activation. W[5] and b[5] are the 5th layer parameters.

  • Superscript (i) denotes an object associated with the ith example.

    • Example: x(i) is the ith training example input.

  • Superscript ⟨t⟩ denotes an object at the tth time-step.

    • Example: x⟨t⟩ is the input x at the tth time-step. x(i)⟨t⟩ is the input at the tth timestep of example i.

  • Lowerscript i denotes the ith entry of a vector.

    • Example: a[l]i denotes the ith entry of the activations in layer l.

We assume that you are already familiar with numpy and/or have completed the previous courses of the specialization. Let's get started!

 

Let's first import all the packages that you will need during this assignment.

In [96]: 
import numpy as np
from rnn_utils import *

1 - Forward propagation for the basic Recurrent Neural Network

Later this week, you will generate music using an RNN. The basic RNN that you will implement has the structure below. In this example, Tx=Ty.


Figure 1: Basic RNN model

Here's how you can implement an RNN:

Steps:

  1. Implement the calculations needed for one time-step of the RNN.
  2. Implement a loop over Tx time-steps in order to process all the inputs, one at a time.

Let's go!

1.1 - RNN cell

A Recurrent neural network can be seen as the repetition of a single cell. You are first going to implement the computations for a single time-step. The following figure describes the operations for a single time-step of an RNN cell.

Figure 2: Basic RNN cell. Takes as input  x⟨t⟩ (current input) and a⟨t−1⟩ (previous hidden state containing information from the past), and outputs a⟨t⟩ which is given to the next RNN cell and also used to predict y⟨t⟩

Exercise: Implement the RNN-cell described in Figure (2).

Instructions:

  1. Compute the hidden state with tanh activation: a⟨t⟩=tanh(Waaa⟨t−1⟩+Waxx⟨t⟩+ba).
  2. Using your new hidden state a⟨t⟩, compute the prediction ˆy⟨t⟩=softmax(Wyaa⟨t⟩+by). We provided you a function: softmax.
  3. Store (a⟨t⟩,a⟨t−1⟩,x⟨t⟩,parameters) in cache
  4. Return a⟨t⟩ , y⟨t⟩ and cache

We will vectorize over m examples. Thus, x⟨t⟩ will have dimension (nx,m), and a⟨t⟩ will have dimension (na,m).

In [97]: 
# GRADED FUNCTION: rnn_cell_forward
​
def rnn_cell_forward(xt, a_prev, parameters):
    """
    Implements a single forward step of the RNN-cell as described in Figure (2)
​
    Arguments:
    xt -- your input data at timestep "t", numpy array of shape (n_x, m).
    a_prev -- Hidden state at timestep "t-1", numpy array of shape (n_a, m)
    parameters -- python dictionary containing:
                        Wax -- Weight matrix multiplying the input, numpy array of shape (n_a, n_x)
                        Waa -- Weight matrix multiplying the hidden state, numpy array of shape (n_a, n_a)
                        Wya -- Weight matrix relating the hidden-state to the output, numpy array of shape (n_y, n_a)
                        ba --  Bias, numpy array of shape (n_a, 1)
                        by -- Bias relating the hidden-state to the output, numpy array of shape (n_y, 1)
    Returns:
    a_next -- next hidden state, of shape (n_a, m)
    yt_pred -- prediction at timestep "t", numpy array of shape (n_y, m)
    cache -- tuple of values needed for the backward pass, contains (a_next, a_prev, xt, parameters)
    """
    
    # Retrieve parameters from "parameters"
    Wax = parameters["Wax"]
    Waa = parameters["Waa"]
    Wya = parameters["Wya"]
    ba = parameters["ba"]
    by = parameters["by"]
    
    ### START CODE HERE ### (≈2 lines)
    # compute next activation state using the formula given above
    a_next = np.tanh(np.dot(Waa,a_prev)+np.dot(Wax,xt)+ba)
    # compute output of the current cell using the formula given above
    yt_pred = softmax(np.dot(Wya,a_next)+by) 
    ### END CODE HERE ###
    
    # store values you need for backward propagation in cache
    cache = (a_next, a_prev, xt, parameters)
    
    return a_next, yt_pred, cache
In [98]: 
np.random.seed(1)
xt = np.random.randn(3,10)
a_prev = np.random.randn(5,10)
Waa = np.random.randn(5,5)
Wax = np.random.randn(5,3)
Wya = np.random.randn(2,5)
ba = np.random.randn(5,1)
by = np.random.randn(2,1)
parameters = {"Waa": Waa, "Wax": Wax, "Wya": Wya, "ba": ba, "by": by}
​
a_next, yt_pred, cache = rnn_cell_forward(xt, a_prev, parameters)
print("a_next[4] = ", a_next[4])
print("a_next.shape = ", a_next.shape)
print("yt_pred[1] =", yt_pred[1])
print("yt_pred.shape = ", yt_pred.shape)
a_next[4] =  [ 0.59584544  0.18141802  0.61311866  0.99808218  0.85016201  0.99980978
 -0.18887155  0.99815551  0.6531151   0.82872037]
a_next.shape =  (5, 10)
yt_pred[1] = [ 0.9888161   0.01682021  0.21140899  0.36817467  0.98988387  0.88945212
  0.36920224  0.9966312   0.9982559   0.17746526]
yt_pred.shape =  (2, 10)

Expected Output:

a_next[4]:[ 0.59584544 0.18141802 0.61311866 0.99808218 0.85016201 0.99980978 -0.18887155 0.99815551 0.6531151 0.82872037]
a_next.shape:(5, 10)
yt[1]:[ 0.9888161 0.01682021 0.21140899 0.36817467 0.98988387 0.88945212 0.36920224 0.9966312 0.9982559 0.17746526]
yt.shape:(2, 10)
 

1.2 - RNN forward pass

You can see an RNN as the repetition of the cell you've just built. If your input sequence of data is carried over 10 time steps, then you will copy the RNN cell 10 times. Each cell takes as input the hidden state from the previous cell (a⟨t−1⟩) and the current time-step's input data (x⟨t⟩). It outputs a hidden state (a⟨t⟩) and a prediction (y⟨t⟩) for this time-step.

Figure 3: Basic RNN. The input sequence  x=(x⟨1⟩,x⟨2⟩,...,x⟨Tx⟩) is carried over Tx time steps. The network outputs y=(y⟨1⟩,y⟨2⟩,...,y⟨Tx⟩).

Exercise: Code the forward propagation of the RNN described in Figure (3).

Instructions:

  1. Create a vector of zeros (a) that will store all the hidden states computed by the RNN.
  2. Initialize the "next" hidden state as a0 (initial hidden state).
  3. Start looping over each time step, your incremental index is t :
    • Update the "next" hidden state and the cache by running rnn_cell_forward
    • Store the "next" hidden state in a (tth position)
    • Store the prediction in y
    • Add the cache to the list of caches
  4. Return a, y and caches
In [99]: 
# GRADED FUNCTION: rnn_forward
​
def rnn_forward(x, a0, parameters):
    """
    Implement the forward propagation of the recurrent neural network described in Figure (3).
​
    Arguments:
    x -- Input data for every time-step, of shape (n_x, m, T_x).
    a0 -- Initial hidden state, of shape (n_a, m)
    parameters -- python dictionary containing:
                        Waa -- Weight matrix multiplying the hidden state, numpy array of shape (n_a, n_a)
                        Wax -- Weight matrix multiplying the input, numpy array of shape (n_a, n_x)
                        Wya -- Weight matrix relating the hidden-state to the output, numpy array of shape (n_y, n_a)
                        ba --  Bias numpy array of shape (n_a, 1)
                        by -- Bias relating the hidden-state to the output, numpy array of shape (n_y, 1)
​
    Returns:
    a -- Hidden states for every time-step, numpy array of shape (n_a, m, T_x)
    y_pred -- Predictions for every time-step, numpy array of shape (n_y, m, T_x)
    caches -- tuple of values needed for the backward pass, contains (list of caches, x)
    """
    
    # Initialize "caches" which will contain the list of all caches
    caches = []
    
    # Retrieve dimensions from shapes of x and parameters["Wya"]
    n_x, m, T_x = x.shape
    n_y, n_a = parameters["Wya"].shape
    
    ### START CODE HERE ###
    
    # initialize "a" and "y" with zeros (≈2 lines)
    a = np.zeros((n_a,m,T_x))
    y_pred = np.zeros((n_y,m,T_x))
    
    # Initialize a_next (≈1 line)
    a_next = a0
    
    # loop over all time-steps
    for t in range(T_x):
        # Update next hidden state, compute the prediction, get the cache (≈1 line)
        a_next, yt_pred, cache = rnn_cell_forward(x[:,:,t],a_next,parameters)
        # Save the value of the new "next" hidden state in a (≈1 line)
        a[:,:,t] = a_next
        # Save the value of the prediction in y (≈1 line)
        y_pred[:,:,t] = yt_pred
        # Append "cache" to "caches" (≈1 line)
        caches.append(cache)
        
    ### END CODE HERE ###
    
    # store values needed for backward propagation in cache
    caches = (caches, x)
    
    return a, y_pred, caches
In [100]: 
np.random.seed(1)
x = np.random.randn(3,10,4)
a0 = np.random.randn(5,10)
Waa = np.random.randn(5,5)
Wax = np.random.randn(5,3)
Wya = np.random.randn(2,5)
ba = np.random.randn(5,1)
by = np.random.randn(2,1)
parameters = {"Waa": Waa, "Wax": Wax, "Wya": Wya, "ba": ba, "by": by}
​
a, y_pred, caches = rnn_forward(x, a0, parameters)
print("a[4][1] = ", a[4][1])
print("a.shape = ", a.shape)
print("y_pred[1][3] =", y_pred[1][3])
print("y_pred.shape = ", y_pred.shape)
print("caches[1][1][3] =", caches[1][1][3])
print("len(caches) = ", len(caches))
a[4][1] =  [-0.99999375  0.77911235 -0.99861469 -0.99833267]
a.shape =  (5, 10, 4)
y_pred[1][3] = [ 0.79560373  0.86224861  0.11118257  0.81515947]
y_pred.shape =  (2, 10, 4)
caches[1][1][3] = [-1.1425182  -0.34934272 -0.20889423  0.58662319]
len(caches) =  2

Expected Output:

a[4][1]:[-0.99999375 0.77911235 -0.99861469 -0.99833267]
a.shape:(5, 10, 4)
y[1][3]:[ 0.79560373 0.86224861 0.11118257 0.81515947]
y.shape:(2, 10, 4)
cache[1][1][3]:[-1.1425182 -0.34934272 -0.20889423 0.58662319]
len(cache):2
 

Congratulations! You've successfully built the forward propagation of a recurrent neural network from scratch. This will work well enough for some applications, but it suffers from vanishing gradient problems. So it works best when each output y⟨t⟩ can be estimated using mainly "local" context (meaning information from inputs x⟨t′⟩ where t′ is not too far from t).

In the next part, you will build a more complex LSTM model, which is better at addressing vanishing gradients. The LSTM will be better able to remember a piece of information and keep it saved for many timesteps.

 

2 - Long Short-Term Memory (LSTM) network

This following figure shows the operations of an LSTM-cell.

Figure 4: LSTM-cell. This tracks and updates a "cell state" or memory variable  c⟨t⟩ at every time-step, which can be different from a⟨t⟩.

Similar to the RNN example above, you will start by implementing the LSTM cell for a single time-step. Then you can iteratively call it from inside a for-loop to have it process an input with Tx time-steps.

About the gates

- Forget gate

For the sake of this illustration, let's assume we are reading words in a piece of text, and want use an LSTM to keep track of grammatical structures, such as whether the subject is singular or plural. If the subject changes from a singular word to a plural word, we need to find a way to get rid of our previously stored memory value of the singular/plural state. In an LSTM, the forget gate let's us do this:

Here, Wf are weights that govern the forget gate's behavior. We concatenate [a⟨t−1⟩,x⟨t⟩] and multiply by Wf. The equation above results in a vector Γ⟨t⟩f with values between 0 and 1. This forget gate vector will be multiplied element-wise by the previous cell state c⟨t−1⟩. So if one of the values of Γ⟨t⟩f is 0 (or close to 0) then it means that the LSTM should remove that piece of information (e.g. the singular subject) in the corresponding component of c⟨t−1⟩. If one of the values is 1, then it will keep the information.

- Update gate

Once we forget that the subject being discussed is singular, we need to find a way to update it to reflect that the new subject is now plural. Here is the formula for the update gate:

 

Similar to the forget gate, here Γ⟨t⟩u is again a vector of values between 0 and 1. This will be multiplied element-wise with ˜c⟨t⟩, in order to compute c⟨t⟩.

- Updating the cell

To update the new subject we need to create a new vector of numbers that we can add to our previous cell state. The equation we use is:

Finally, the new cell state is:

 
- Output gate

To decide which outputs we will use, we will use the following two formulas:

Where in equation 5 you decide what to output using a sigmoid function and in equation 6 you multiply that by the tanh of the previous state.

 

2.1 - LSTM cell

Exercise: Implement the LSTM cell described in the Figure (4).

Instructions:

  1. Concatenate a⟨t−1⟩ and x⟨t⟩ in a single matrix: 

     

  2. Compute all the formulas 1-6. You can use sigmoid() (provided) and np.tanh().
  3. Compute the prediction y⟨t⟩. You can use softmax() (provided).
In [101]: 
# GRADED FUNCTION: lstm_cell_forward
​
def lstm_cell_forward(xt, a_prev, c_prev, parameters):
    """
    Implement a single forward step of the LSTM-cell as described in Figure (4)
​
    Arguments:
    xt -- your input data at timestep "t", numpy array of shape (n_x, m).
    a_prev -- Hidden state at timestep "t-1", numpy array of shape (n_a, m)
    c_prev -- Memory state at timestep "t-1", numpy array of shape (n_a, m)
    parameters -- python dictionary containing:
                        Wf -- Weight matrix of the forget gate, numpy array of shape (n_a, n_a + n_x)
                        bf -- Bias of the forget gate, numpy array of shape (n_a, 1)
                        Wi -- Weight matrix of the update gate, numpy array of shape (n_a, n_a + n_x)
                        bi -- Bias of the update gate, numpy array of shape (n_a, 1)
                        Wc -- Weight matrix of the first "tanh", numpy array of shape (n_a, n_a + n_x)
                        bc --  Bias of the first "tanh", numpy array of shape (n_a, 1)
                        Wo -- Weight matrix of the output gate, numpy array of shape (n_a, n_a + n_x)
                        bo --  Bias of the output gate, numpy array of shape (n_a, 1)
                        Wy -- Weight matrix relating the hidden-state to the output, numpy array of shape (n_y, n_a)
                        by -- Bias relating the hidden-state to the output, numpy array of shape (n_y, 1)
    Returns:
    a_next -- next hidden state, of shape (n_a, m)
    c_next -- next memory state, of shape (n_a, m)
    yt_pred -- prediction at timestep "t", numpy array of shape (n_y, m)
    cache -- tuple of values needed for the backward pass, contains (a_next, c_next, a_prev, c_prev, xt, parameters)
    Note: ft/it/ot stand for the forget/update/output gates, cct stands for the candidate value (c tilde),
          c stands for the memory value
    """
​
    # Retrieve parameters from "parameters"
    Wf = parameters["Wf"]
    bf = parameters["bf"]
    Wi = parameters["Wi"]
    bi = parameters["bi"]
    Wc = parameters["Wc"]
    bc = parameters["bc"]
    Wo = parameters["Wo"]
    bo = parameters["bo"]
    Wy = parameters["Wy"]
    by = parameters["by"]
    
    # Retrieve dimensions from shapes of xt and Wy
    n_x, m = xt.shape
    n_y, n_a = Wy.shape
​
    ### START CODE HERE ###
    # Concatenate a_prev and xt (≈3 lines)
    concat = np.zeros((n_a+n_x,m))
    concat[: n_a, :] = a_prev
    concat[n_a :, :] = xt
​
    # Compute values for ft, it, cct, c_next, ot, a_next using the formulas given figure (4) (≈6 lines)
    ft = sigmoid(np.dot(Wf,concat)+bf)
    it = sigmoid(np.dot(Wi,concat)+bi)
    cct = np.tanh(np.dot(Wc,concat)+bc)
    c_next = ft*c_prev + it*cct
    ot = sigmoid(np.dot(Wo,concat)+bo)
    a_next = ot*np.tanh(c_next)
    
    # Compute prediction of the LSTM cell (≈1 line)
    yt_pred = softmax(np.dot(Wy,a_next)+by)
    ### END CODE HERE ###
​
    # store values needed for backward propagation in cache
    cache = (a_next, c_next, a_prev, c_prev, ft, it, cct, ot, xt, parameters)
​
    return a_next, c_next, yt_pred, cache
In [102]: 
np.random.seed(1)
xt = np.random.randn(3,10)
a_prev = np.random.randn(5,10)
c_prev = np.random.randn(5,10)
Wf = np.random.randn(5, 5+3)
bf = np.random.randn(5,1)
Wi = np.random.randn(5, 5+3)
bi = np.random.randn(5,1)
Wo = np.random.randn(5, 5+3)
bo = np.random.randn(5,1)
Wc = np.random.randn(5, 5+3)
bc = np.random.randn(5,1)
Wy = np.random.randn(2,5)
by = np.random.randn(2,1)
​
parameters = {"Wf": Wf, "Wi": Wi, "Wo": Wo, "Wc": Wc, "Wy": Wy, "bf": bf, "bi": bi, "bo": bo, "bc": bc, "by": by}
​
a_next, c_next, yt, cache = lstm_cell_forward(xt, a_prev, c_prev, parameters)
print("a_next[4] = ", a_next[4])
print("a_next.shape = ", c_next.shape)
print("c_next[2] = ", c_next[2])
print("c_next.shape = ", c_next.shape)
print("yt[1] =", yt[1])
print("yt.shape = ", yt.shape)
print("cache[1][3] =", cache[1][3])
print("len(cache) = ", len(cache))
a_next[4] =  [-0.66408471  0.0036921   0.02088357  0.22834167 -0.85575339  0.00138482
  0.76566531  0.34631421 -0.00215674  0.43827275]
a_next.shape =  (5, 10)
c_next[2] =  [ 0.63267805  1.00570849  0.35504474  0.20690913 -1.64566718  0.11832942
  0.76449811 -0.0981561  -0.74348425 -0.26810932]
c_next.shape =  (5, 10)
yt[1] = [ 0.79913913  0.15986619  0.22412122  0.15606108  0.97057211  0.31146381
  0.00943007  0.12666353  0.39380172  0.07828381]
yt.shape =  (2, 10)
cache[1][3] = [-0.16263996  1.03729328  0.72938082 -0.54101719  0.02752074 -0.30821874
  0.07651101 -1.03752894  1.41219977 -0.37647422]
len(cache) =  10

Expected Output:

a_next[4]:[-0.66408471 0.0036921 0.02088357 0.22834167 -0.85575339 0.00138482 0.76566531 0.34631421 -0.00215674 0.43827275]
a_next.shape:(5, 10)
c_next[2]:[ 0.63267805 1.00570849 0.35504474 0.20690913 -1.64566718 0.11832942 0.76449811 -0.0981561 -0.74348425 -0.26810932]
c_next.shape:(5, 10)
yt[1]:[ 0.79913913 0.15986619 0.22412122 0.15606108 0.97057211 0.31146381 0.00943007 0.12666353 0.39380172 0.07828381]
yt.shape:(2, 10)
cache[1][3]:[-0.16263996 1.03729328 0.72938082 -0.54101719 0.02752074 -0.30821874 0.07651101 -1.03752894 1.41219977 -0.37647422]
len(cache):10
 

2.2 - Forward pass for LSTM

Now that you have implemented one step of an LSTM, you can now iterate this over this using a for-loop to process a sequence of Tx inputs.

Figure 5: LSTM over multiple time-steps.

Exercise: Implement lstm_forward() to run an LSTM over Tx time-steps.

Note: c⟨0⟩ is initialized with zeros.

In [103]: 
# GRADED FUNCTION: lstm_forward
​
def lstm_forward(x, a0, parameters):
    """
    Implement the forward propagation of the recurrent neural network using an LSTM-cell described in Figure (4).
​
    Arguments:
    x -- Input data for every time-step, of shape (n_x, m, T_x).
    a0 -- Initial hidden state, of shape (n_a, m)
    parameters -- python dictionary containing:
                        Wf -- Weight matrix of the forget gate, numpy array of shape (n_a, n_a + n_x)
                        bf -- Bias of the forget gate, numpy array of shape (n_a, 1)
                        Wi -- Weight matrix of the update gate, numpy array of shape (n_a, n_a + n_x)
                        bi -- Bias of the update gate, numpy array of shape (n_a, 1)
                        Wc -- Weight matrix of the first "tanh", numpy array of shape (n_a, n_a + n_x)
                        bc -- Bias of the first "tanh", numpy array of shape (n_a, 1)
                        Wo -- Weight matrix of the output gate, numpy array of shape (n_a, n_a + n_x)
                        bo -- Bias of the output gate, numpy array of shape (n_a, 1)
                        Wy -- Weight matrix relating the hidden-state to the output, numpy array of shape (n_y, n_a)
                        by -- Bias relating the hidden-state to the output, numpy array of shape (n_y, 1)
    Returns:
    a -- Hidden states for every time-step, numpy array of shape (n_a, m, T_x)
    y -- Predictions for every time-step, numpy array of shape (n_y, m, T_x)
    caches -- tuple of values needed for the backward pass, contains (list of all the caches, x)
    """
​
    # Initialize "caches", which will track the list of all the caches
    caches = []
    
    ### START CODE HERE ###
    # Retrieve dimensions from shapes of x and parameters['Wy'] (≈2 lines)
    n_x, m, T_x = x.shape
    n_y, n_a = parameters['Wy'].shape
    
    # initialize "a", "c" and "y" with zeros (≈3 lines)
    a = np.zeros((n_a,m,T_x))
    c = np.zeros((n_a,m,T_x))
    y = np.zeros((n_y,m,T_x))
    
    # Initialize a_next and c_next (≈2 lines)
    a_next = a0
    c_next = np.zeros((n_a,m))
    
    # loop over all time-steps
    for t in range(T_x):
        # Update next hidden state, next memory state, compute the prediction, get the cache (≈1 line)
        a_next, c_next, yt, cache = lstm_cell_forward(x[:,:,t], a_next, c_next, parameters)
        # Save the value of the new "next" hidden state in a (≈1 line)
        a[:,:,t] = a_next
        # Save the value of the prediction in y (≈1 line)
        y[:,:,t] = yt
        # Save the value of the next cell state (≈1 line)
        c[:,:,t]  = c_next
        # Append the cache into caches (≈1 line)
        caches.append(cache)
        
    ### END CODE HERE ###
    
    # store values needed for backward propagation in cache
    caches = (caches, x)
​
    return a, y, c, caches
In [104]: 
np.random.seed(1)
x = np.random.randn(3,10,7)
a0 = np.random.randn(5,10)
Wf = np.random.randn(5, 5+3)
bf = np.random.randn(5,1)
Wi = np.random.randn(5, 5+3)
bi = np.random.randn(5,1)
Wo = np.random.randn(5, 5+3)
bo = np.random.randn(5,1)
Wc = np.random.randn(5, 5+3)
bc = np.random.randn(5,1)
Wy = np.random.randn(2,5)
by = np.random.randn(2,1)
​
parameters = {"Wf": Wf, "Wi": Wi, "Wo": Wo, "Wc": Wc, "Wy": Wy, "bf": bf, "bi": bi, "bo": bo, "bc": bc, "by": by}
​
a, y, c, caches = lstm_forward(x, a0, parameters)
print("a[4][3][6] = ", a[4][3][6])
print("a.shape = ", a.shape)
print("y[1][4][3] =", y[1][4][3])
print("y.shape = ", y.shape)
print("caches[1][1[1]] =", caches[1][1][1])
print("c[1][2][1]", c[1][2][1])
print("len(caches) = ", len(caches))
a[4][3][6] =  0.172117767533
a.shape =  (5, 10, 7)
y[1][4][3] = 0.95087346185
y.shape =  (2, 10, 7)
caches[1][1[1]] = [ 0.82797464  0.23009474  0.76201118 -0.22232814 -0.20075807  0.18656139
  0.41005165]
c[1][2][1] -0.855544916718
len(caches) =  2

Expected Output:


a[4][3][6] =0.172117767533
a.shape =(5, 10, 7)
y[1][4][3] =0.95087346185
y.shape =(2, 10, 7)
caches[1][1][1] =[ 0.82797464 0.23009474 0.76201118 -0.22232814 -0.20075807 0.18656139 0.41005165]
c[1][2][1] =-0.855544916718
len(caches) =2
 

Congratulations! You have now implemented the forward passes for the basic RNN and the LSTM. When using a deep learning framework, implementing the forward pass is sufficient to build systems that achieve great performance.

The rest of this notebook is optional, and will not be graded.

 

3 - Backpropagation in recurrent neural networks (OPTIONAL / UNGRADED)

In modern deep learning frameworks, you only have to implement the forward pass, and the framework takes care of the backward pass, so most deep learning engineers do not need to bother with the details of the backward pass. If however you are an expert in calculus and want to see the details of backprop in RNNs, you can work through this optional portion of the notebook.

When in an earlier course you implemented a simple (fully connected) neural network, you used backpropagation to compute the derivatives with respect to the cost to update the parameters. Similarly, in recurrent neural networks you can to calculate the derivatives with respect to the cost in order to update the parameters. The backprop equations are quite complicated and we did not derive them in lecture. However, we will briefly present them below.

 

3.1 - Basic RNN backward pass

We will start by computing the backward pass for the basic RNN-cell.

Figure 6: RNN-cell's backward pass. Just like in a fully-connected neural network, the derivative of the cost function  J backpropagates through the RNN by following the chain-rule from calculus. The chain-rule is also used to calculate (∂J∂Wax,∂J∂Waa,∂J∂b) to update the parameters (Wax,Waa,ba).
Deriving the one step backward functions:

To compute the rnn_cell_backward you need to compute the following equations. It is a good exercise to derive them by hand.

The derivative of tanh is 1−tanh(x)2. You can find the complete proof here. Note that: sech(x)2=1−tanh(x)2

Similarly for ∂a⟨t⟩∂Wax,∂a⟨t⟩∂Waa,∂a⟨t⟩∂b, the derivative of tanh(u) is (1−tanh(u)2)du.

The final two equations also follow same rule and are derived using the tanh derivative. Note that the arrangement is done in a way to get the same dimensions to match.

In [105]: 
def rnn_cell_backward(da_next, cache):
    """
    Implements the backward pass for the RNN-cell (single time-step).
​
    Arguments:
    da_next -- Gradient of loss with respect to next hidden state
    cache -- python dictionary containing useful values (output of rnn_cell_forward())
​
    Returns:
    gradients -- python dictionary containing:
                        dx -- Gradients of input data, of shape (n_x, m)
                        da_prev -- Gradients of previous hidden state, of shape (n_a, m)
                        dWax -- Gradients of input-to-hidden weights, of shape (n_a, n_x)
                        dWaa -- Gradients of hidden-to-hidden weights, of shape (n_a, n_a)
                        dba -- Gradients of bias vector, of shape (n_a, 1)
    """
    
    # Retrieve values from cache
    (a_next, a_prev, xt, parameters) = cache
    
    # Retrieve values from parameters
    Wax = parameters["Wax"]
    Waa = parameters["Waa"]
    Wya = parameters["Wya"]
    ba = parameters["ba"]
    by = parameters["by"]
​
    ### START CODE HERE ###
    # compute the gradient of tanh with respect to a_next (≈1 line)
    dtanh = da_next * (1-a_next**2) #dtanh相当于是∂J/∂a_next * ∂a_next/∂tanh，因为后面所求导数都有这两项，所以单独写出来
​
    # compute the gradient of the loss with respect to Wax (≈2 lines)
    dxt = np.dot(Wax.T, dtanh)
    #dxt = Wax.T * (1-np.tanh(Wax*xt+Waa*a_prev+b)**2)
    dWax = np.dot(dtanh,xt.T)
​
    # compute the gradient with respect to Waa (≈2 lines)
    da_prev = np.dot(Waa.T, dtanh)
    dWaa = np.dot(dtanh,a_prev.T)
​
    # compute the gradient with respect to b (≈1 line)
    dba = np.sum(dtanh, axis=1, keepdims = True) #对列求和，keepdims保持求和后矩阵形状（5，1），否则shape会变成（5，）
​
    ### END CODE HERE ###
    
    # Store the gradients in a python dictionary
    gradients = {"dxt": dxt, "da_prev": da_prev, "dWax": dWax, "dWaa": dWaa, "dba": dba}
    
    return gradients
In [106]: 
np.random.seed(1)
xt = np.random.randn(3,10)
a_prev = np.random.randn(5,10)
Wax = np.random.randn(5,3)
Waa = np.random.randn(5,5)
Wya = np.random.randn(2,5)
b = np.random.randn(5,1)
by = np.random.randn(2,1)
parameters = {"Wax": Wax, "Waa": Waa, "Wya": Wya, "ba": ba, "by": by}
​
a_next, yt, cache = rnn_cell_forward(xt, a_prev, parameters)
​
da_next = np.random.randn(5,10)
gradients = rnn_cell_backward(da_next, cache)
print("gradients[\"dxt\"][1][2] =", gradients["dxt"][1][2])
print("gradients[\"dxt\"].shape =", gradients["dxt"].shape)
print("gradients[\"da_prev\"][2][3] =", gradients["da_prev"][2][3])
print("gradients[\"da_prev\"].shape =", gradients["da_prev"].shape)
print("gradients[\"dWax\"][3][1] =", gradients["dWax"][3][1])
print("gradients[\"dWax\"].shape =", gradients["dWax"].shape)
print("gradients[\"dWaa\"][1][2] =", gradients["dWaa"][1][2])
print("gradients[\"dWaa\"].shape =", gradients["dWaa"].shape)
print("gradients[\"dba\"][4] =", gradients["dba"][4])
print("gradients[\"dba\"].shape =", gradients["dba"].shape)
 
 
 
gradients["dxt"][1][2] = -0.460564103059
gradients["dxt"].shape = (3, 10)
gradients["da_prev"][2][3] = 0.0842968653807
gradients["da_prev"].shape = (5, 10)
gradients["dWax"][3][1] = 0.393081873922
gradients["dWax"].shape = (5, 3)
gradients["dWaa"][1][2] = -0.28483955787
gradients["dWaa"].shape = (5, 5)
gradients["dba"][4] = [ 0.80517166]
gradients["dba"].shape = (5, 1)

Expected Output:

gradients["dxt"][1][2] =-0.460564103059
gradients["dxt"].shape =(3, 10)
gradients["da_prev"][2][3] =0.0842968653807
gradients["da_prev"].shape =(5, 10)
gradients["dWax"][3][1] =0.393081873922
gradients["dWax"].shape =(5, 3)
gradients["dWaa"][1][2] =-0.28483955787
gradients["dWaa"].shape =(5, 5)
gradients["dba"][4] =[ 0.80517166]
gradients["dba"].shape =(5, 1)
 
Backward pass through the RNN

Computing the gradients of the cost with respect to a⟨t⟩ at every time-step t is useful because it is what helps the gradient backpropagate to the previous RNN-cell. To do so, you need to iterate through all the time steps starting at the end, and at each step, you increment the overall dba, dWaa, dWax and you store dx.

Instructions:

Implement the rnn_backward function. Initialize the return variables with zeros first and then loop through all the time steps while calling the rnn_cell_backward at each time timestep, update the other variables accordingly.

In [107]: 
def rnn_backward(da, caches):
    """
    Implement the backward pass for a RNN over an entire sequence of input data.
​
    Arguments:
    da -- Upstream gradients of all hidden states, of shape (n_a, m, T_x)
    caches -- tuple containing information from the forward pass (rnn_forward)
    Returns:
    gradients -- python dictionary containing:
                        dx -- Gradient w.r.t. the input data, numpy-array of shape (n_x, m, T_x)
                        da0 -- Gradient w.r.t the initial hidden state, numpy-array of shape (n_a, m)
                        dWax -- Gradient w.r.t the input's weight matrix, numpy-array of shape (n_a, n_x)
                        dWaa -- Gradient w.r.t the hidden state's weight matrix, numpy-arrayof shape (n_a, n_a)
                        dba -- Gradient w.r.t the bias, of shape (n_a, 1)
    """
        
    ### START CODE HERE ###
    
    # Retrieve values from the first cache (t=1) of caches (≈2 lines)
    (caches, x) = caches #参照前面rnn_forward
    (a1, a0, x1, parameters) = caches[0] #第一个cache对应的下标为0
    
    # Retrieve dimensions from da's and x1's shapes (≈2 lines)
    n_a, m, T_x = da.shape
    n_x, m = x1.shape
    
    # initialize the gradients with the right sizes (≈6 lines)
    dx = np.zeros((n_x,m,T_x))
    dWax = np.zeros((n_a,n_x))
    dWaa = np.zeros((n_a,n_a))
    dba = np.zeros((n_a,1))
    da0 = np.zeros((n_a,m))
    da_prevt = np.zeros((n_a,m))
    
    # Loop through all the time steps
    for t in reversed(range(T_x)): #这里注意是reversed，因为是反向传播所以是从第t层->第1层
        # Compute gradients at time step t. Choose wisely the "da_next" and the "cache" to use in the backward propagation step. (≈1 line)
        gradients = rnn_cell_backward(da[:,:,t]+da_prevt, caches[t])
        #这里为什么参数da_next = da[:,:,t]+da_prevt我没有完全理解，我的理解是反向传播计算第t步的da时，
        #输入第t步的da加上第t步后面累加的da（因为是反向传播所以后面传过来的da在上一步循环中已经计算出啦了，是da_prevt）作为da_next
        #da_prevt作为一个中间变量用来记录第t步之后累加的da值，所以向前走到第0步时，da_prevt就记录了累加到最后的结果，并令da0=da_prevt保存这个结果
        # Retrieve derivatives from gradients (≈ 1 line)
        dxt, da_prevt, dWaxt, dWaat, dbat = gradients["dxt"], gradients["da_prev"], gradients["dWax"], gradients["dWaa"], gradients["dba"]
        # Increment global derivatives w.r.t parameters by adding their derivative at time-step t (≈4 lines)
        dx[:, :, t] = dxt
        dWax += dWaxt
        dWaa += dWaat
        dba += dbat
        
    # Set da0 to the gradient of a which has been backpropagated through all time-steps (≈1 line) 
    da0 = da_prevt
    ### END CODE HERE ###
​
    # Store the gradients in a python dictionary
    gradients = {"dx": dx, "da0": da0, "dWax": dWax, "dWaa": dWaa,"dba": dba}
    
    return gradients
In [108]: 
np.random.seed(1)
x = np.random.randn(3,10,4)
a0 = np.random.randn(5,10)
Wax = np.random.randn(5,3)
Waa = np.random.randn(5,5)
Wya = np.random.randn(2,5)
ba = np.random.randn(5,1)
by = np.random.randn(2,1)
parameters = {"Wax": Wax, "Waa": Waa, "Wya": Wya, "ba": ba, "by": by}
a, y, caches = rnn_forward(x, a0, parameters)
da = np.random.randn(5, 10, 4)
gradients = rnn_backward(da, caches)
​
print("gradients[\"dx\"][1][2] =", gradients["dx"][1][2])
print("gradients[\"dx\"].shape =", gradients["dx"].shape)
print("gradients[\"da0\"][2][3] =", gradients["da0"][2][3])
print("gradients[\"da0\"].shape =", gradients["da0"].shape)
print("gradients[\"dWax\"][3][1] =", gradients["dWax"][3][1])
print("gradients[\"dWax\"].shape =", gradients["dWax"].shape)
print("gradients[\"dWaa\"][1][2] =", gradients["dWaa"][1][2])
print("gradients[\"dWaa\"].shape =", gradients["dWaa"].shape)
print("gradients[\"dba\"][4] =", gradients["dba"][4])
print("gradients[\"dba\"].shape =", gradients["dba"].shape)
 
 
 
gradients["dx"][1][2] = [-2.07101689 -0.59255627  0.02466855  0.01483317]
gradients["dx"].shape = (3, 10, 4)
gradients["da0"][2][3] = -0.314942375127
gradients["da0"].shape = (5, 10)
gradients["dWax"][3][1] = 11.2641044965
gradients["dWax"].shape = (5, 3)
gradients["dWaa"][1][2] = 2.30333312658
gradients["dWaa"].shape = (5, 5)
gradients["dba"][4] = [-0.74747722]
gradients["dba"].shape = (5, 1)

Expected Output:

gradients["dx"][1][2] =[-2.07101689 -0.59255627 0.02466855 0.01483317]
gradients["dx"].shape =(3, 10, 4)
gradients["da0"][2][3] =-0.314942375127
gradients["da0"].shape =(5, 10)
gradients["dWax"][3][1] =11.2641044965
gradients["dWax"].shape =(5, 3)
gradients["dWaa"][1][2] =2.30333312658
gradients["dWaa"].shape =(5, 5)
gradients["dba"][4] =[-0.74747722]
gradients["dba"].shape =(5, 1)
 

3.2 - LSTM backward pass

 

3.2.1 One Step backward

The LSTM backward pass is slighltly more complicated than the forward one. We have provided you with all the equations for the LSTM backward pass below. (If you enjoy calculus exercises feel free to try deriving these from scratch yourself.)

3.2.2 gate derivatives

3.2.3 parameter derivatives

To calculate dbf,dbu,dbc,dbo you just need to sum across the horizontal (axis= 1) axis on dΓ⟨t⟩f,dΓ⟨t⟩u,d˜c⟨t⟩,dΓ⟨t⟩o respectively. Note that you should have the keep_dims = True option.

Finally, you will compute the derivative with respect to the previous hidden state, previous memory state, and input.

Here, the weights for equations 13 are the first n_a, (i.e. Wf=Wf[:na,:] etc...)

转载于:https://www.cnblogs.com/RB26DETT/p/11444943.html

weixin_30780221
关注
Building a Recurrent Neural Network Step by Step
dh0029314的博客
02-07 5036
Building your Recurrent Neural Network - Step by Step Welcome to Course 5’s first assignment! In this assignment, you will implement your first Recurrent Neural Network in numpy. Recurrent Neural Ne...
Coursera 吴恩达DeepLearning.AI 第五课 sequence model 序列模型 第一周Building your Recurrent Neural Network - Ste
forqzy的专栏
02-06 3093
本周的习题有点多，主要是python不熟悉，然后时间不够，提醒说马上过期才开始看的视频，optional部分没有写完Building your Recurrent Neural Network - Step by StepWelcome to Course 5's first assignment! In this assignment, you will implement your first...
吴恩达RNN作业:Building your Recurrent Neural Network - Step by Step
08-06
吴恩达RNN作业:Building your Recurrent Neural Network - Step by Step
Building a Recurrent Neural Network - Step by Step
05-27
Andrew Ng在Coursera上的RNN练习,第一周第一节内容,完整未作过的练习. 注意:不包含答案的.
neural networks deep learning Building your deep neural network: Step by Step Homework
热门推荐
生物医学信息学博客
08-29 1万+
Building your Deep Neural Network: Step by Step Welcome to your week 4 assignment (part 1 of 2)! You have previously trained a 2-layer Neural Network (with a single hidden layer). This week, you will build a de
[coursera/SequenceModels/week1]Building a Recurrent Neural Network - Step by Step - v3[assignment]
gdtop的个人笔记
02-15 7718
Building your Recurrent Neural Network - Step by StepWelcome to Course 5's first assignment! In this assignment, you will implement your first Recurrent Neural Network in numpy.Recurrent Neural Networ...
Sequence Level Training with Recurrent Neural Networks-学习笔记
jiayoudangdang的博客
02-21 1567
强化学习NLP的论文…… Sequence Level Training with Recurrent Neural Networks 用递归神经网络进行序列级训练 0摘要 许多自然语言处理应用程序使用语言模型来生成文本。这些模型通常被训练来预测序列中的下一个单词，给定前面的单词和一些上下文，比如图像。然而，在测试时，模型将从头开始生成整个序列。这种差异使生成变得脆弱，因为错误可能会在生成过程中累积。我们通过提出一种新的序列级训练算法来解决这个问题，该算法直接优化测试时使用的度量，例如BLEU或RO
A Critical Review of Recurrent Neural Networks for Sequence Learning
02-13
A Critical Review of Recurrent Neural Networks for Sequence Learning
【11】Fast and accurate recurrent neural network acoustic models
08-26
RNNs can be equaled by sequence trained phone models initialized with connectionist temporal classification (CTC). In this paper, we present techniques that further improve performance of LSTM RNN ...
构建RNN、LSTM
Harder_14的博客
09-11 318
1.导入包 import numpy as np from rnn_utils import * 2.构建RNN def rnn_cell_forward(xt, a_prev, parameters): """ Implements a single forward step of the RNN-cell as described in Figure (2) Arguments: xt -- your input data at timestep "t",
16.深度学习练习：Building your Recurrent Neural Network - Step by Step
10点43的博客
11-09 422
Building your Recurrent Neural Network - Step by Step1 - Forward propagation for the basic Recurrent Neural Network1.1 - RNN cell1.2 - RNN forward pass2 - Long Short-Term Memory (LSTM) network- Forget...
[Building your Recurrent Neural Network - Step by Step] [题目错误]
nkhgl的博客
08-09 246
在DeepLearning.ai的第五门课Sequence Models的第一周课程的第一个作业Building your Recurrent Neural Network - Step by Step出现了一个题目错误，先粘贴原题目代码： def rnn_backward(da, caches): """ Implement the backward pass for a R...
深度学习 course5 week1 Building+a+Recurrent+Neural+Network+-+Step+by+Step
qq_31119155的博客
07-16 605
  def rnn_cell_forward(xt,a_prev,parameters): Wax=parameters['Wax'] Waa=parameters['Waa'] Wya=parameters['Wya'] ba=parameters['ba'] by=parameters['by'] # a_next=np.tanh(np....
Assignment | 05-week1 -Building your Recurrent Neural Network - Step by Step
Self Improvement Lab
02-28 2014
该系列仅在原课程基础上课后作业部分添加个人学习笔记，如有错误，还请批评指教。- ZJ Coursera 课程 |deeplearning.ai |网易云课堂 CSDN：http://blog.csdn.net/JUNJUN_ZHAO/article/details/79400107 Welcome to Course 5’s first assignment! In this a...
【吴恩达深度学习】Building your Recurrent Neural Network - Step by Step
无
04-03 1429
Building your Recurrent Neural Network - Step by Step Welcome to Course 5’s first assignment! In this assignment, you will implement your first Recurrent Neural Network in numpy. Recurrent Neural Networks (RNN) are very effective for Natural Language Proce
coursera 吴恩达 -- 第五课 RNN ：第一周Building a Recurrent Neural Network - Step by Step - v3
qq_25436597的博客
04-03 3065
强烈建议 不要写选做部分！！！
深度学习作业L5W1（1）：Building your Recurrent Neural Network - Step by Step
awake020
05-25 239
利用numpy实现RNN、LSTM的前向传播和反向传播。主要是为了熟悉基本单元的流程 RNN 基本单元： # GRADED FUNCTION: rnn_cell_forward def rnn_cell_forward(xt, a_prev, parameters): """ Implements a single forward step of the RNN-cell as described in Figure (2) Arguments: xt -- your i
吴恩达 深度学习 序列模型 第一周 编程作业一 手动建立RNN LSTM 总结
csdn_inside的博客
02-20 1044
Assignment 1 : Building a recurrent neural network - step by step 1.RNN Cell 这个函数用来计算RNN Cell。RNN Cell的结构和公式如下： 输入有三个：一个是该时刻的序列x<t>，一个是上一次传过来的信息激励a<t-1>。 parameters用于获取Cell的参数Wax、Waa...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值