Coursera深度学习课程 DeepLearning.ai 编程作业——Character level language model - Dinosaurus land

Character level language model - Dinosaurus land

Welcome to Dinosaurus Island! 65 million years ago, dinosaurs existed, and in this assignment they are back. You are in charge of a special task. Leading biology researchers are creating new breeds of dinosaurs and bringing them to life on earth, and your job is to give names to these dinosaurs. If a dinosaur does not like its name, it might go beserk, so choose wisely!

Luckily you have learned some deep learning and you will use it to save the day. Your assistant has collected a list of all the dinosaur names they could find, and compiled them into this dataset. (Feel free to take a look by clicking the previous link.) To create new dinosaur names, you will build a character level language model to generate new names. Your algorithm will learn the different name patterns, and randomly generate new names. Hopefully this algorithm will keep you and your team safe from the dinosaurs’ wrath!

By completing this assignment you will learn:

• How to store text data for processing using an RNN
• How to synthesize(合成) data, by sampling predictions at each time step and passing it to the next RNN-cell unit
• How to build a character-level text generation recurrent neural network
• Why clipping the gradients is important

We will begin by loading in some functions that we have provided for you in rnn_utils. Specifically, you have access to functions such as rnn_forward and rnn_backward which are equivalent to those you’ve implemented in the previous assignment.

import numpy as np
from utils import *
import random

1 - Problem Statement

1.1 - Dataset and Preprocessing

Run the following cell to read the dataset of dinosaur names, create a list of unique characters (such as a-z), and compute the dataset and vocabulary size.

data = open('dinos.txt', 'r').read()
data= data.lower()
chars = list(set(data))  #a=set('google')  print(a)-> set(['e','o','g','l'])  print(list(a))->['e','o','g','l']  
data_size, vocab_size = len(data), len(chars)
print('There are %d total characters and %d unique characters in your data.' % (data_size, vocab_size))

Output:

There are 19910 total characters and 27 unique characters in your data.

The characters are a-z (26 characters) plus the “\n” (or newline character), which in this assignment plays a role similar to the <EOS> (or “End of sentence”) token we had discussed in lecture, only here it indicates the end of the dinosaur name rather than the end of a sentence. In the cell below, we create a python dictionary (i.e., a hash table) to map each character to an index from 0-26（索引1：每个字母对应索引）. We also create a second python dictionary that maps each index back to the corresponding character character（索引2：每个索引对应相应的字母）. This will help you figure out what index corresponds to what character in the probability distribution output of the softmax layer (这将帮助你了解在softmax层的概率分布输出中哪些索引对应哪些字母). Below, char_to_ix and ix_to_char are the python dictionaries.

char_to_ix = { ch:i for i,ch in enumerate(sorted(chars)) }  #sorted(chars) 按 \n,a-z顺序进行排序
ix_to_char = { i:ch for i,ch in enumerate(sorted(chars)) }
print(ix_to_char)

Output:

{0: '\n', 1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z'}

1.2 - Overview of the model

Your model will have the following structure:

• Initialize parameters
• Run the optimization loop
  • Forward propagation to compute the loss function
  • Backward propagation to compute the gradients with respect to the loss function
  • Clip the gradients to avoid exploding gradients(梯度爆炸)
  • Using the gradients, update your parameter with the gradient descent update rule.
• Return the learned parameters

**Figure 1**: Recurrent Neural Network, similar to what you had built in the previous notebook "Building a RNN - Step by Step".

At each time-step, the RNN tries to predict what is the next character given the previous characters. The dataset $X=(x^{⟨1⟩},x^{⟨2⟩},...,x^{⟨T_x⟩})$ is a list of characters in the training set, while $Y=(y^{⟨1⟩},y^{⟨2⟩},...,y^{⟨T_x⟩})$ is such that at every time-step $t$, we have $y^{⟨t⟩}=x^{⟨t+1⟩}$.

2 - Building blocks of the model

In this part, you will build two important blocks of the overall model:

• Gradient clipping: to avoid exploding gradients
• Sampling: a technique used to generate characters

You will then apply these two functions to build the model.

2.1 - Clipping the gradients in the optimization loop

In this section you will implement the clip function that you will call inside of your optimization loop. Recall that your overall loop structure usually consists of a forward pass, a cost computation, a backward pass, and a parameter update. Before updating the parameters, you will perform gradient clipping when needed to make sure that your gradients are not “exploding,” meaning taking on overly large values.

(在这个章节中，你会执行‘clip’函数，然后在optimization loop中被调用。代码的循环结构经常包含forward pass，cost computation，backward pass，parameter update。在更新参数之前，你将会展示gradient clipping来保证你的梯度不会爆炸，梯度爆炸是指出现非常大的值。)

In the exercise below, you will implement a function clip that takes in a dictionary of gradients and returns a clipped version of gradients if needed. There are different ways to clip gradients; we will use a simple element-wise clipping procedure, in which every element of the gradient vector is clipped to lie between some range [-N, N]. More generally, you will provide a maxValue (say 10). In this example, if any component of the gradient vector is greater than 10, it would be set to 10; and if any component of the gradient vector is less than -10, it would be set to -10. If it is between -10 and 10, it is left alone.

（在下面的练习中，你将会执行函数‘clip’,这个函数包含一个梯度的字典，然后返回梯度的裁剪版本。有很多办法来裁剪梯度，我们将使用简单的element-wise裁剪流程，每一个梯度向量的元素被裁剪，使得其值位于[-N,N]之间。更一般地，你将会提供一个‘maxValue’（比如说设为10）。在这个例子中，如果梯度向量的任何一个值大于10，将会被设为10,。如果梯度向量中任何一部分比-10小，这部分将会被设为-10。

**Figure 2**: Visualization of gradient descent with and without gradient clipping, in a case where the network is running into slight "exploding gradient" problems.

Exercise: Implement the function below to return the clipped gradients of your dictionary gradients. Your function takes in a maximum threshold and returns the clipped versions of your gradients. You can check out this hint for examples of how to clip in numpy. You will need to use the argument out = ....

（执行下面的函数来返回裁剪过的梯度，你的函数包含一个最大阈值和返回裁剪版的梯度。你可以对照这个链接作为例子来知道怎么在numpy执行裁剪，你需要使用参数‘out= …’）

def clip(gradients, maxValue):
    '''
    Clips the gradients' values between minimum and maximum.
    
    Arguments:
    gradients -- a dictionary containing the gradients "dWaa", "dWax", "dWya", "db", "dby"
    maxValue -- everything above this number is set to this number, and everything less than -maxValue is set to -maxValue
    
    Returns: 
    gradients -- a dictionary with the clipped gradients.
    '''
    
    dWaa, dWax, dWya, db, dby = gradients['dWaa'], gradients['dWax'], gradients['dWya'], gradients['db'], gradients['dby']
   
    ### START CODE HERE ###
    # clip to mitigate exploding gradients, loop over [dWax, dWaa, dWya, db, dby]. (≈2 lines)
    for gradient in [dWax, dW