Chapter 7 Neural Networks and Neural Language Models

Chapter 7 Neural Networks and Neural Language Models

Speech and Language Processing ed3 读书笔记

McCulloch-Pitts neuron (McCulloch and Pitts, 1943)

feed-forward network: the computation proceeds iteratively from one layer of units to the next.

deep learning, because modern networks are often deep (have many layers) z

7.1 Units

The building block of a neural network is a single computational unit. A unit takes a set of real valued numbers as input, performs some computation on them, and produces an output.

Given a input vector $x$, a unit has a weight vector $w$ and a scalar bias $b$, so the weighted sum $z$ can
be represented as:
$$z=w\cdot x+b$$
Finally, instead of using $z$, a linear function of $x$, as the output, neural units apply a non-linear function $f$ to $z$. We will refer to the output of this function as the activation value for the unit, $a$. Since we are just modeling a single unit, the activation for the node is in fact the final output of the network, which we’ll generally call $y$. So the value $y$ is defined as:
$$y=a=f(z)$$
Three popular non-linear functions $f()$ are the sigmoid, the tanh, and the rectified linear ReLU.
$$y=\sigma (z)=\frac{1}{1+e^{-z}}$$

$$y=\tanh (z)=\frac{e^z-e^{-z}}{e^z+e^{-z}}$$

$$y=\max (z,0)$$

These activation functions have different properties that make them useful for different language applications or network architectures. For example the rectifier function has nice properties that result from it being very close to linear. In the sigmoid or tanh functions, very high values of $z$ result in values of $y$ that are saturated, i.e., extremely close to 1, which causes problems for learning. Rectifiers don’t have this problem, since the output of values close to 1 also approaches 1 in a nice gentle linear way. By contrast, the tanh function has the nice properties of being smoothly differentiable and mapping outlier values toward the mean.

7.2 The XOR problem

It turns out, however, that it’s not possible to build a perceptron to compute logical XOR!

The intuition behind this important result relies on understanding that a perceptron is a linear classifier.

We say that XOR is not a linearly separable function.

7.2.1 The solution: neural networks

While the XOR function cannot be calculated by a single perceptron, it can be calculated by a layered network of units. Goodfellow et al. (2016) compute XOR using two layers of ReLU-based units.

7.3 Feed-Forward Neural Networks

A feed-forward network is a multilayer network in which the units are connected with no cycles; the outputs from units in each layer are passed to units in the next higher layer, and no outputs are passed back to lower layers. (In Chapter 9 we’ll introduce networks with cycles, called recurrent neural networks).

Simple feed-forward networks have three kinds of nodes: input units, hidden units, and output units.

The core of the neural network is the hidden layer formed of hidden units, each of which is a neural unit, taking a weighted sum of its inputs and then applying a non-linearity. In the standard architecture, each layer is fully-connected, meaning that each unit in each layer takes as input the outputs from all the units in the previous layer, and there is a link between every pair of units from two adjacent layers. Thus each hidden unit sums over all the input units.

We represent the parameters for the entire hidden layer by combining the weight vector ${\text{w}}_i$ and bias $ b_i$ for each hidden unit $h_i$ into a single weight matrix $W$ and a single bias vector $b$ for the whole layer. Each element $W_{ij}$ of the weight matrix $W$ represents the weight of the connection from the $j$th input unit $x_j$ to the $i$th hidden unit $h_i$.

The advantage of using a single matrix $W$ for the weights of the entire layer is that hidden layer computation for a feedforward network can be done very efficiently with simple matrix operations. In fact, the computation only has three steps: multiplying the weight matrix $W$ by the input vector $x$, adding the bias vector $b$, and applying the activation function $g$ (such as the sigmoid, tanh, or relu activation function defined above).

The output of the hidden layer, the vector $h$, is thus the following, using the sigmoid function $\sigma$:
$$h=\sigma (Wx+b)$$
Let’s introduce some constants to represent the dimensionalities of these vectors and matrices. We’ll refer to the input layer as layer 0 of the network, and use $n_0$ to represent the dimension of a input, so $x$ is a vector of real numbers of dimension $n_0$, or more formally $x\in {\mathbb{R}}^{n_0}$. Let’s call the hidden layer layer 1 and the output layer layer 2. The hidden layer has dimensionality $n_1$, so $h\in {\mathbb{R}}^{n_1}$ and also $b\in {\mathbb{R}}^{n_1}$ (since each hidden unit can take a different bias value). And the weight matrix $W$ has dimensionality $W\in {\mathbb{R}}^{n_1\times n_0}$.

$h_i=w_i\cdot x+b_i={\sum }_{j=1}^{n_0}{w}_{ij}{x}_j+b_i$.

Like the hidden layer, the output layer has a weight matrix (let’s call it $U$), but output layers may not have a bias vector $b$, so we’ll simplify by eliminating the bias vector in this example. The weight matrix is multiplied by its input vector ($h$) to produce the intermediate output $z$.
$$z=Uh$$
There are $n_2$ output nodes, so $z\in {\mathbb{R}}^{n_2}$, weight matrix $U$ has dimensionality $U\in {\mathbb{R}}^{n_2\times n_1}$, and element $U_{ij}$ is the weight from unit $h_j$ in the hidden layer to unit $i$ in the output layer.

However, $z$ can’t be the output of the classifier, since it’s a vector of real-valued numbers, while what we need for classification is a vector of probabilities. There is a convenient function for normalizing a vector of real values, by which we mean converting it to a vector that encodes a probability distribution (all the numbers lie between 0 and 1 and sum to 1): the softmax function. For a vector $z$ of dimensionality $d$, the softmax is defined as:
$$\text{softmax}(z_i)=\frac{e^{z_i}}{{\sum }_{j=1}^d{e}^{z_j}}$$
A neural network is like logistic regression, but (a) with many layers, since a deep neural network is like layer after layer of logistic regression classifiers, and (b) rather than forming the features by feature templates, the prior layers of the network induce the feature representations themselves.

Let’s now set up some notation to make it easier to talk about deeper networks of depth more than 2. We’ll use superscripts in square brackets to mean layer numbers, starting at 0 for the input layer. So $W^{[1]}$ will mean the weight matrix for the (first) hidden layer, and $b^{[1]}$ will mean the bias vector for the (first) hidden layer. $n_j$ will mean the number of units at layer $j$. We’ll use $g(\cdot )$ to stand for the activation function, which will tend to be ReLU or tanh for intermediate layers and softmax for output layers. We’ll use $a^{[i]}$ to mean the output from layer $i$, and $z^{[i]}$ to mean the combination of weights and biases $W^{[i]}{a}^{[i-1]}+b^{[i]}$. The 0th layer is for inputs, so the inputs $x$ we’ll refer to more generally as $a^{[0]}$.

Thus we’ll represent a 3-layer net as follows:
KaTeX parse error: Unknown column alignment: [ at position 15: \begin{array}[̲ll]\ z^{[1]}=& …
activation functions $g(\cdot )$ are generally different at the final layer. Thus $g^{[2]}$ might be softmax for multinomial classification or sigmoid for binary classification, while ReLU or tanh might be the activation function $g()$ at the internal layers.

7.4 Training Neural Nets

First, we’ll need a loss function that models the distance between the system output and the gold output, and it’s common to use the loss used for logistic regression, the cross-entropy loss. Second, to find the parameters that minimize this loss function, we’ll use the gradient descent optimization algorithm. Third, gradient descent requires knowing the gradient of the loss function, the vector that contains the partial derivative of the loss function with respect to each of the parameters. The answer is the algorithm called error back-propagation or reverse differentiation.

7.4.1 Loss function

Let $y$ be a vector over the $K$ classes representing the true output probability distribution. The cross entropy loss here is
$$L_{CE}(\hat{y},y)=-\sum _{i=1}^K{y}_i\log {\hat{y}}_i$$
We can simplify this equation further. Assume this is a hard classification task, meaning that only one class is the correct one, and that there is one output unit in $y$ for each class. If the true class is $i$, then $y$ is a vector where $y_i=1$ and $y_j=0,\forall j̸=i$. A vector like this, with one value=1 and the rest 0, is called a one-hot vector. Now let $\hat{y}$ be the vector output from the network. The sum in Eq. 10 will be 0 except for the true class. Hence the cross-entropy loss is simply the log probability of the likelihood loss negative log correct class, and we therefore also call this the negative log likelihood loss:
$$L_{CE}(\hat{y},y)=-\log {\hat{y}}_i=-\log \frac{e^{z_i}}{{\sum }_{j=1}^K{e}^{z_j}}$$

7.4.2 Computing the Gradient

For a network with one weight layer and sigmoid output (which is what logistic regression is),
$$\frac{\partial L_{CE}(w,b)}{\partial w_j}=(\hat{y}-y)x_j=(\sigma (w\cdot x+b)-y)x_j$$
Or for a network with one hidden layer and softmax output,
∂ L C E ∂ w k = ( 1 { y = k } − p ( y = k ∣ x ) ) x k = ( 1 { y = k } − e w k ⋅ x + b k ∑ j = 1 K e w j ⋅ x + b j ) x k \frac{\partial L_{CE}}{\partial w_k}=(1\{y=k\}- p(y=k|x))x_k=\left(1\{y=k\}-\frac{e^{w_k\cdot x+b_k}}{\sum_{j=1}^K e^{w_j\cdot x+b_j}}\right)x_k ∂wk​∂LCE​​=(1{y=k}−p(y=k∣x))xk​=(1{y=k}−∑j=1K​ewj​⋅x+bj​ewk​⋅x+bk​​)xk​
But these derivatives only give correct updates for one weight layer: the last one! For deep networks, computing the gradients for each weight is much more complex, since we are computing the derivative with respect to weight parameters that appear all the way back in the very early layers of the network, even though the loss is computed only at the very end of the network.

The solution to computing this gradient is an algorithm called error backpropagation or backprop (Rumelhart et al., 1986). While backprop was invented specially for neural networks, it turns out to be the same as a more general procedure called backward differentiation, which depends on the notion of computation graphs.

7.4.3 Computation Graphs

A computation graph is a representation of the process of computing a mathematical expression, in which the computation is broken down into separate operations, each of which is modeled as a node in a graph. In a computation graph, each node represents a operation, and each directed edge represents the output from a operation as the input to the next.

7.4.4 Backward differentiation on computation graphs

The importance of the computation graph comes from the backward pass, which is used to compute the derivatives that we’ll need for the weight update. Backwards differentiation makes use of the chain rule in calculus. In the backward pass, we compute each of these partials along each edge of the graph from right to left, multiplying the necessary partials to result in the final derivative we need. At each node we need to compute the local partial derivative with respect to the parent, multiply it by the partial derivative that is being passed down from the parent, and then pass it to the child.
$$\frac{d\sigma (z)}{dz}=\sigma (z)(1-z)$$

$$\frac{d\tanh (z)}{dz}=1-{\tanh }^2(z)$$

KaTeX parse error: Unknown column alignment: [ at position 52: …\{\begin{array}[̲ll]\ 0& \textrm…

7.4.5 More details on learning

Optimization in neural networks is a non-convex optimization problem, more complex than for logistic regression, and for that and other reasons there are many best practices for successful learning.

For logistic regression we can initialize gradient descent with all the weights and biases having the value 0. In neural networks, by contrast, we need to initialize the weights with small random numbers. It’s also helpful to normalize the input values to have 0 mean and unit variance.

Various forms of regularization are used to prevent overfitting. One of the most important is dropout: randomly dropping some units and their connections from the network during training (Hinton et al. 2012, Srivastava et al. 2014).

Hyperparameter tuning is also important. The parameters of a neural network are the weights W and biases b; those are learned by gradient descent. The hyperparameters are things that are set by the algorithm designer and not learned in the same way, although they must be tuned. Hyperparameters include the learning rate $\eta$, the minibatch size, the model architecture (the number of layers, the number of hidden nodes per layer, the choice of activation functions), how to regularize, and so on. Gradient descent itself also has many architectural variants such as Adam (Kingma and Ba, 2015).

Finally, most modern neural networks are built using computation graph formalisms that make all the work of gradient computation and parallelization onto vector-based GPUs (Graphic Processing Units) very easy and natural. Pytorch (Paszke et al., 2017) and TensorFlow (Abadi et al., 2015) are two of the most popular.

7.5 Neural Language Models

Neural net-based language models turn out to have many advantages over the ngram language models of Chapter 3. Among these are that neural language models don’t need smoothing, they can handle much longer histories, and they can generalize over contexts of similar words. For a training set of a given size, a neural language model has much higher predictive accuracy than an n-gram language model. Furthermore, neural language models underlie many of the models we’ll introduce
for tasks like machine translation, dialog, and language generation.

On the other hand, there is a cost for this improved performance: neural net language models are strikingly slower to train than traditional language models, and so for many tasks an n-gram language model is still the right tool.

In this chapter we’ll describe simple feedforward neural language models, first introduced by Bengio et al. (2003). Modern neural language models are generally not feedforward but recurrent, using the technology that we will introduce in Chapter 9.

A feedforward neural LM is a standard feedforward network that takes as input at time $t$ a representation of some number of previous words ($w_{t-1}$,$w_{t-2}$, etc) and outputs a probability distribution over possible next words. Thus—like the n-gram LM—the feedforward neural LM approximates the probability of a word given the entire prior context $P(w_t|w_1^{t-1})$ by approximating based on the $N$ previous words:
$$P(w_t|w_1^{t-1})\approx P(w_t|w_{t-N+1}^{t-1})$$
In the following examples we’ll use a 4-gram example, so we’ll show a net to estimate the probability $P(w_t=i|w_{t-1},w_{t-2},w_{t-3})$.

7.5.1 Embeddings

In neural language models, the prior context is represented by embeddings of the previous words. Representing the prior context as embeddings, rather than by exact words as used in n-gram language models, allows neural language models to generalize to unseen data much better than n-gram language models.

Let’s see how this works in practice. Let’s assume we have an embedding dictionary $E$ that gives us, for each word in our vocabulary $V$ , the embedding for that word, perhaps precomputed by an algorithm like word2vec from Chapter 6.

Fig. 7.12 shows a sketch of this simplified FFNNLM with N=3; we have a moving window at time $t$ with an embedding vector representing each of the 3 previous words (words $w_{t-1}$,$w_{t-2}$,and $w_{t-3}$). These 3 vectors are concatenated together to produce $x$, the input layer of a neural network whose output is a softmax with a probability distribution over words. Thus $y_{42}$, the value of output node 42 is the probability of the next word $w_t$ being $V_{42}$, the vocabulary word with index 42.

The model shown in Fig. 7.12 is quite sufficient, assuming we learn the embeddings separately by a method like the word2vec methods of Chapter 6. The method of using another algorithm to learn the embedding representations we use for input pretraining words is called pretraining. If those pretrained embeddings are sufficient for your purposes, then this is all you need.

However, often we’d like to learn the embeddings simultaneously with training the network. This is true when whatever task the network is designed for (sentiment classification, or translation, or parsing) places strong constraints on what makes a good representation.

Let’s therefore show an architecture that allows the embeddings to be learned. To do this, we’ll add an extra layer to the network, and propagate the error all the way back to the embedding vectors, starting with embeddings with random values and slowly moving toward sensible representations.

For this to work at the input layer, instead of pre-trained embeddings, we’re going to represent each of the $N$ previous words as a one-hot vector of length $|V|$, i.e., one-hot vector with one dimension for each word in the vocabulary.

Fig. 7.13 shows the additional layers needed to learn the embeddings during LM training. Here the N=3 context words are represented as 3 one-hot vectors, fully connected to the embedding layer via 3 instantiations of the $E$ embedding matrix. Note that we don’t want to learn separate weight matrices for mapping each of the 3 previous words to the projection layer, we want one single embedding dictionary $E$ that’s shared among these three. That’s because over time, many different words will appear as $w_{t-2}$ or $w_{t-1}$, and we’d like to just represent each word with one vector, whichever context position it appears in. The embedding weight matrix $E$ thus has a row for each word, each a vector of $d$ dimensions (the dimension of an embedding), and hence has dimensionality $V\times d$.

Let’s walk through the forward pass of Fig. 7.13.

1. Select three embeddings from E: Given the three previous words, we look up their indices, create 3 one-hot vectors, and then multiply each by the embedding matrix $E$. Consider $w_{t-3}$. The one-hot vector for ‘the’ (index 35) is multiplied by the embedding matrix $E$, to give the first part of the first hidden layer, called the projection layer. Since each row of the project layer $e$ is just an embedding for a word, and the input is a one-hot columnvector $x_i$ for word $V_i$, the projection layer for input $x_i$ will be $E{x}_i=e_i$, the embedding for word $V_i$. We now concatenate the three embeddings for the context words.
2. Multiply by W: We now multiply by $W$ (and add $b$) and pass through the rectified linear (or other) activation function to get the hidden layer $h$.
3. Multiply by U: $h$ is now multiplied by $U$
4. Apply softmax: After the softmax, each node $y_i$ in the output layer estimates the probability $P(w_t=i|w_{t-1},w_{t-2},w_{t-3})$

In summary, if we use $e$ to represent the projection layer, formed by concatenating the 3 embedding for the three context vectors, the equations for a neural language model become:
KaTeX parse error: Unknown column alignment: [ at position 15: \begin{array}[̲lll]\ e&=&(Ex_1…

7.5.2 Training the neural language model

To train the model, i.e. to set all the parameters $\theta =E,W,U,b$, we do gradient descent, using error back propagation on the computation graph to compute the gradient.

Generally, training proceeds by taking as input a very long text, concatenating all the sentences, start with random weights, and then iteratively moving through the text predicting each word $w_t$. At each word $w_t$, the cross-entropy (negative log likelihood) loss is:
$$L=-\log p(w_t|w_{t-1},\dots ,w_{t-n+1})$$
The gradient for this loss is then:
$${\theta }_{t+1}={\theta }_t-\eta \frac{\partial -\log p(w_t|w_{t-1},\dots ,w_{t-n+1})}{\partial \theta }$$
Training the parameters to minimize loss will result both in an algorithm for language modeling (a word predictor) but also a new set of embeddings $e$ that can be used as word representations for other tasks.

7.6 Summary

• Neural networks are built out of neural units, originally inspired by human neurons but now simple an abstract computational device.
• Each neural unit multiplies input values by a weight vector, adds a bias, and then applies a non-linear activation function like sigmoid, tanh, or rectified linear.
• In a fully-connected, feedforward network, each unit in layer $i$ is connected to each unit in layer $i+1$, and there are no cycles.
• The power of neural networks comes from the ability of early layers to learn representations that can be utilized by later layers in the network.
• Neural networks are trained by optimization algorithms like gradient descent.
• Error back propagation, backward differentiation on a computation graph, is used to compute the gradients of the loss function for a network.
• Neural language models use a neural network as a probabilistic classifier, to compute the probability of the next word given the previous $n$ words.
• Neural language models can use pretrained embeddings, or can learn embeddings from scratch in the process of language modeling.