CHAPTER 5 Why are deep neural networks hard to train?

Imagine you're an engineer who has been asked to design a computerfrom scratch. One day you're working away in your office, designinglogical circuits, setting out AND gates, OR gates,and so on, when your boss walks in with bad news. The customer hasjust added a surprising design requirement: the circuit for the entirecomputer must be just two layers deep:

You're dumbfounded, and tell your boss: "The customer is crazy!"

Your boss replies: "I think they're crazy, too. But what thecustomer wants, they get."

In fact, there's a limited sense in which the customer isn't crazy.Suppose you're allowed to use a special logical gate which lets youAND together as many inputs as you want. And you're alsoallowed a many-input NAND gate, that is, a gate which canAND multiple inputs and then negate the output. With thesespecial gates it turns out to be possible to compute any function atall using a circuit that's just two layers deep.

But just because something is possible doesn't make it a good idea.In practice, when solving circuit design problems (or most any kind ofalgorithmic problem), we usually start by figuring out how to solvesub-problems, and then gradually integrate the solutions. In otherwords, we build up to a solution through multiple layers ofabstraction.

For instance, suppose we're designing a logical circuit to multiplytwo numbers. Chances are we want to build it up out of sub-circuitsdoing operations like adding two numbers. The sub-circuits for addingtwo numbers will, in turn, be built up out of sub-sub-circuits foradding two bits. Very roughly speaking our circuit will look like:

That is, our final circuit contains at least three layers of circuitelements. In fact, it'll probably contain more than three layers, aswe break the sub-tasks down into smaller units than I've described.But you get the general idea.

So deep circuits make the process of design easier. But they're notjust helpful for design. There are, in fact, mathematical proofsshowing that for some functions very shallow circuits requireexponentially more circuit elements to compute than do deep circuits.For instance, a famous series of papers in the early1980s**The history is somewhat complex, so I won't give detailed references. See Johan Håstad's 2012 paper On the correlation of parity and small-depth circuits for an account of the early history and references. showed that computing the parity of a setof bits requires exponentially many gates, if done with a shallowcircuit. On the other hand, if you use deeper circuits it's easy tocompute the parity using a small circuit: you just compute the parityof pairs of bits, then use those results to compute the parity ofpairs of pairs of bits, and so on, building up quickly to the overallparity. Deep circuits thus can be intrinsically much more powerfulthan shallow circuits.

Up to now, this book has approached neural networks like the crazycustomer. Almost all the networks we've worked with have just asingle hidden layer of neurons (plus the input and output layers):

These simple networks have been remarkably useful: in earlier chapterswe used networks like this to classify handwritten digits with betterthan 98 percent accuracy! Nonetheless, intuitively we'd expectnetworks with many more hidden layers to be more powerful:

Such networks could use the intermediate layers to build up multiplelayers of abstraction, just as we do in Boolean circuits. Forinstance, if we're doing visual pattern recognition, then the neuronsin the first layer might learn to recognize edges, the neurons in thesecond layer could learn to recognize more complex shapes, saytriangle or rectangles, built up from edges. The third layer wouldthen recognize still more complex shapes. And so on. These multiplelayers of abstraction seem likely to give deep networks a compellingadvantage in learning to solve complex pattern recognition problems.Moreover, just as in the case of circuits, there are theoreticalresults suggesting that deep networks are intrinsically more powerfulthan shallow networks**For certain problems and network architectures this is proved in On the number of response regions of deep feed forward networks with piece-wise linear activations, by Razvan Pascanu, Guido Montúfar, and Yoshua Bengio (2014). See also the more informal discussion in section 2 of Learning deep architectures for AI, by Yoshua Bengio (2009)..

How can we train such deep networks? In this chapter, we'll trytraining deep networks using our workhorse learning algorithm -stochastic gradient descent by backpropagation. But we'llrun into trouble, with our deep networks not performing much (if atall) better than shallow networks.

That failure seems surprising in the light of the discussion above.Rather than give up on deep networks, we'll dig down and try tounderstand what's making our deep networks hard to train. When welook closely, we'll discover that the different layers in our deepnetwork are learning at vastly different speeds. In particular, whenlater layers in the network are learning well, early layers often getstuck during training, learning almost nothing at all. This stucknessisn't simply due to bad luck. Rather, we'll discover there arefundamental reasons the learning slowdown occurs, connected to our useof gradient-based learning techniques.

As we delve into the problem more deeply, we'll learn that theopposite phenomenon can also occur: the early layers may be learningwell, but later layers can become stuck. In fact, we'll find thatthere's an intrinsic instability associated to learning by gradientdescent in deep, many-layer neural networks. This instability tendsto result in either the early or the later layers getting stuck duringtraining.

This all sounds like bad news. But by delving into thesedifficulties, we can begin to gain insight into what's required totrain deep networks effectively. And so these investigations are goodpreparation for the next chapter, where we'll use deep learning toattack image recognition problems.

The vanishing gradient problem

So, what goes wrong when we try to train a deep network?

To answer that question, let's first revisit the case of a networkwith just a single hidden layer. As per usual, we'll use the MNISTdigit classification problem as our playground for learning andexperimentation**I introduced the MNIST problem and data here and here..

If you wish, you can follow along by training networks on yourcomputer. It is also, of course, fine to just read along. If you dowish to follow live, then you'll need Python 2.7, Numpy, and a copy ofthe code, which you can get by cloning the relevant repository fromthe command line:

git clone https://github.com/mnielsen/neural-networks-and-deep-learning.git

If you don't use git then you can download the data and code here. You'll need to change into the src subdirectory.

Then, from a Python shell we load the MNIST data:

>>> import mnist_loader
>>> training_data, validation_data, test_data = \
... mnist_loader.load_data_wrapper()

We set up our network:

>>> import network2
>>> net = network2.Network([784, 30, 10])

This network has 784 neurons in the input layer, corresponding to the 28×28=784

pixels in the input image. We use 30 hiddenneurons, as well as 10 output neurons, corresponding to the 10possible classifications for the MNIST digits ('0', '1', '2', …

,'9').

Let's try training our network for 30 complete epochs, usingmini-batches of 10 training examples at a time, a learning rate η=0.1

, and regularization parameter λ=5.0

. As we trainwe'll monitor the classification accuracy on thevalidation_data**Note that the networks is likely to take some minutes to train, depending on the speed of your machine. So if you're running the code you may wish to continue reading and return later, not wait for the code to finish executing.:

>>> net.SGD(training_data, 30, 10, 0.1, lmbda=5.0, 
... evaluation_data=validation_data, monitor_evaluation_accuracy=True)

We get a classification accuracy of 96.48 percent (or thereabouts -it'll vary a bit from run to run), comparable to our earlier resultswith a similar configuration.

Now, let's add another hidden layer, also with 30 neurons in it, andtry training with the same hyper-parameters:

>>> net = network2.Network([784, 30, 30, 10])
>>> net.SGD(training_data, 30, 10, 0.1, lmbda=5.0, 
... evaluation_data=validation_data, monitor_evaluation_accuracy=True)

This gives an improved classification accuracy, 96.90 percent. That'sencouraging: a little more depth is helping. Let's add another30-neuron hidden layer:

>>> net = network2.Network([784, 30, 30, 30, 10])
>>> net.SGD(training_data, 30, 10, 0.1, lmbda=5.0, 
... evaluation_data=validation_data, monitor_evaluation_accuracy=True)

That doesn't help at all. In fact, the result drops back down to96.57 percent, close to our original shallow network. And suppose weinsert one further hidden layer:

>>> net = network2.Network([784, 30, 30, 30, 30, 10])
>>> net.SGD(training_data, 30, 10, 0.1, lmbda=5.0, 
... evaluation_data=validation_data, monitor_evaluation_accuracy=True)

The classification accuracy drops again, to 96.53 percent. That'sprobably not a statistically significant drop, but it's notencouraging, either.

This behaviour seems strange. Intuitively, extra hidden layers oughtto make the network able to learn more complex classificationfunctions, and thus do a better job classifying. Certainly, thingsshouldn't get worse, since the extra layers can, in the worst case,simply do nothing**See this later problem to understand how to build a hidden layer that does nothing.. But that's not what's going on.

So what is going on? Let's assume that the extra hidden layers reallycould help in principle, and the problem is that our learningalgorithm isn't finding the right weights and biases. We'd like tofigure out what's going wrong in our learning algorithm, and how to dobetter.

To get some insight into what's going wrong, let's visualize how thenetwork learns. Below, I've plotted part of a [784,30,30,10]

network, i.e., a network with two hidden layers, each containing 30 hidden neurons. Each neuron in the diagram has a little bar on it,representing how quickly that neuron is changing as the networklearns. A big bar means the neuron's weights and bias are changingrapidly, while a small bar means the weights and bias are changingslowly. More precisely, the bars denote the gradient ∂C/∂b

for each neuron, i.e., the rate of change of the costwith respect to the neuron's bias. Back inChapter 2 we saw that this gradient quantity controlled not just howrapidly the bias changes during learning, but also how rapidly theweights input to the neuron change, too. Don't worry if you don'trecall the details: the thing to keep in mind is simply that thesebars show how quickly each neuron's weights and bias are changing asthe network learns.

To keep the diagram simple, I've shown just the top six neurons in thetwo hidden layers. I've omitted the input neurons, since they've gotno weights or biases to learn. I've also omitted the output neurons,since we're doing layer-wise comparisons, and it makes most sense tocompare layers with the same number of neurons. The results areplotted at the very beginning of training, i.e., immediately after thenetwork is initialized. Here they are**The data plotted is generated using the program generate_gradient.py. The same program is also used to generate the results quoted later in this section.:

The network was initialized randomly, and so it's not surprising thatthere's a lot of variation in how rapidly the neurons learn. Still,one thing that jumps out is that the bars in the second hidden layerare mostly much larger than the bars in the first hidden layer. As aresult, the neurons in the second hidden layer will learn quite a bitfaster than the neurons in the first hidden layer. Is this merely acoincidence, or are the neurons in the second hidden layer likely tolearn faster than neurons in the first hidden layer in general?

To determine whether this is the case, it helps to have a global wayof comparing the speed of learning in the first and second hiddenlayers. To do this, let's denote the gradient as δlj=∂C/∂blj

, i.e., the gradient for the j th neuron in the l thlayer* *Back in Chapter 2 we referred to this as the error, but here we'll adopt the informal term "gradient". I say "informal" because of course this doesn't explicitly include the partial derivatives of the cost with respect to the weights, ∂C/∂w .. We canthink of the gradient δ1 as a vector whose entries determinehow quickly the first hidden layer learns, and δ2 as a vectorwhose entries determine how quickly the second hidden layer learns.We'll then use the lengths of these vectors as (rough!) globalmeasures of the speed at which the layers are learning. So, forinstance, the length ∥δ1∥ measures the speed at which thefirst hidden layer is learning, while the length ∥δ2∥

measures the speed at which the second hidden layer is learning.

With these definitions, and in the same configuration as was plottedabove, we find ∥δ1∥=0.07…

and ∥δ2∥=0.31…

. So this confirms our earlier suspicion: the neurons inthe second hidden layer really are learning much faster than theneurons in the first hidden layer.

What happens if we add more hidden layers? If we have three hiddenlayers, in a [784,30,30,30,10]

network, then the respectivespeeds of learning turn out to be 0.012, 0.060, and 0.283. Again,earlier hidden layers are learning much slower than later hiddenlayers. Suppose we add yet another layer with 30

hidden neurons.In that case, the respective speeds of learning are 0.003, 0.017,0.070, and 0.285. The pattern holds: early layers learn slower thanlater layers.

We've been looking at the speed of learning at the start of training,that is, just after the networks are initialized. How does the speedof learning change as we train our networks? Let's return to look atthe network with just two hidden layers. The speed of learningchanges as follows:

To generate these results, I used batch gradient descent with just1,000 training images, trained over 500 epochs. This is a bitdifferent than the way we usually train - I've used no mini-batches,and just 1,000 training images, rather than the full 50,000 imagetraining set. I'm not trying to do anything sneaky, or pull the woolover your eyes, but it turns out that using mini-batch stochasticgradient descent gives much noisier (albeit very similar, when youaverage away the noise) results. Using the parameters I've chosen isan easy way of smoothing the results out, so we can see what's goingon.

In any case, as you can see the two layers start out learning at verydifferent speeds (as we already know). The speed in both layers thendrops very quickly, before rebounding. But through it all, the firsthidden layer learns much more slowly than the second hidden layer.

What about more complex networks? Here's the results of a similarexperiment, but this time with three hidden layers (a [784,30,30,30,10]

network):

Again, early hidden layers learn much more slowly than later hiddenlayers. Finally, let's add a fourth hidden layer (a [784,30,30,30,30,10]

network), and see what happens when we train:

Again, early hidden layers learn much more slowly than later hiddenlayers. In this case, the first hidden layer is learning roughly 100times slower than the final hidden layer. No wonder we were havingtrouble training these networks earlier!

We have here an important observation: in at least some deep neuralnetworks, the gradient tends to get smaller as we move backwardthrough the hidden layers. This means that neurons in the earlierlayers learn much more slowly than neurons in later layers. And whilewe've seen this in just a single network, there are fundamentalreasons why this happens in many neural networks. The phenomenon isknown as the vanishing gradient problem**See Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, by Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, and Jürgen Schmidhuber (2001). This paper studied recurrent neural nets, but the essential phenomenon is the same as in the feedforward networks we are studying. See also Sepp Hochreiter's earlier Diploma Thesis, Untersuchungen zu dynamischen neuronalen Netzen (1991, in German)..

Why does the vanishing gradient problem occur? Are there ways we canavoid it? And how should we deal with it in training deep neuralnetworks? In fact, we'll learn shortly that it's not inevitable,although the alternative is not very attractive, either: sometimes thegradient gets much larger in earlier layers! This is theexploding gradient problem, and it's not much better news thanthe vanishing gradient problem. More generally, it turns out that thegradient in deep neural networks is unstable, tending to eitherexplode or vanish in earlier layers. This instability is afundamental problem for gradient-based learning in deep neuralnetworks. It's something we need to understand, and, if possible,take steps to address.

One response to vanishing (or unstable) gradients is to wonder ifthey're really such a problem. Momentarily stepping away from neuralnets, imagine we were trying to numerically minimize a function f(x)

of a single variable. Wouldn't it be good news if the derivative f′(x)

was small? Wouldn't that mean we were already near anextremum? In a similar way, might the small gradient in early layersof a deep network mean that we don't need to do much adjustment of theweights and biases?

Of course, this isn't the case. Recall that we randomly initializedthe weight and biases in the network. It is extremely unlikely ourinitial weights and biases will do a good job at whatever it is wewant our network to do. To be concrete, consider the first layer ofweights in a [784,30,30,30,10]

network for the MNIST problem.The random initialization means the first layer throws away mostinformation about the input image. Even if later layers have beenextensively trained, they will still find it extremely difficult toidentify the input image, simply because they don't have enoughinformation. And so it can't possibly be the case that not muchlearning needs to be done in the first layer. If we're going to traindeep networks, we need to figure out how to address the vanishinggradient problem.

What's causing the vanishing gradient problem? Unstable gradients in deep neural nets

To get insight into why the vanishing gradient problem occurs, let'sconsider the simplest deep neural network: one with just a singleneuron in each layer. Here's a network with three hidden layers:

Here, w1,w2,…

are the weights, b1,b2,… are thebiases, and C is some cost function. Just to remind you how thisworks, the output aj from the j th neuron is σ(zj) , where σ is the usual sigmoid activation function, and zj=wjaj−1+bj is the weightedinput to the neuron. I've drawn the cost C at the end to emphasizethat the cost is a function of the network's output, a4

: if theactual output from the network is close to the desired output, thenthe cost will be low, while if it's far away, the cost will be high.

We're going to study the gradient ∂C/∂b1

associated to the first hidden neuron. We'll figure out an expressionfor ∂C/∂b1

, and by studying that expression we'llunderstand why the vanishing gradient problem occurs.

I'll start by simply showing you the expression for ∂C/∂b1

. It looks forbidding, but it's actually got a simplestructure, which I'll describe in a moment. Here's the expression(ignore the network, for now, and note that σ′ is just thederivative of the σ

function):

The structure in the expression is as follows: there is a σ′(zj)

term in the product for each neuron in the network; aweight wj term for each weight in the network; and a final ∂C/∂a4

term, corresponding to the cost functionat the end. Notice that I've placed each term in the expression abovethe corresponding part of the network. So the network itself is amnemonic for the expression.

You're welcome to take this expression for granted, and skip to thediscussion of how it relates to the vanishing gradient problem. There's no harm in doing this, since theexpression is a special case of ourearlier discussion of backpropagation. But there's also a simpleexplanation of why the expression is true, and so it's fun (andperhaps enlightening) to take a look at that explanation.

Imagine we make a small change Δb1

in the bias b1 . Thatwill set off a cascading series of changes in the rest of the network.First, it causes a change Δa1 in the output from the firsthidden neuron. That, in turn, will cause a change Δz2 in theweighted input to the second hidden neuron. Then a change Δa2 in the output from the second hidden neuron. And so on, all theway through to a change ΔC in the cost at the output. We have

∂C∂b1≈ΔCΔb1.(114)

This suggests that we can figure out an expression for the gradient ∂C/∂b1

by carefully tracking the effect of eachstep in this cascade.

To do this, let's think about how Δb1

causes the output a1 from the first hidden neuron to change. We have a1=σ(z1)=σ(w1a0+b1) , so

Δa1≈=∂σ(w1a0+b1)∂b1Δb1σ′(z1)Δb1.(115)(116)

That σ′(z1) term should look familiar: it's the first term inour claimed expression for the gradient ∂C/∂b1 .Intuitively, this term converts a change Δb1 in the bias intoa change Δa1 in the output activation. That change Δa1 in turn causes a change in the weighted input z2=w2a1+b2 to the second hidden neuron:

Δz2≈=∂z2∂a1Δa1w2Δa1.(117)(118)

Combining our expressions for Δz2 and Δa1 , we seehow the change in the bias b1 propagates along the network toaffect z2 :

Δz2≈σ′(z1)w2Δb1.(119)

Again, that should look familiar: we've now got the first two terms inour claimed expression for the gradient ∂C/∂b1

.

We can keep going in this fashion, tracking the way changes propagatethrough the rest of the network. At each neuron we pick up a σ′(zj)

term, and through each weight we pick up a wj term.The end result is an expression relating the final change ΔC in cost to the initial change Δb1 in the bias:

ΔC≈σ′(z1)w2σ′(z2)…σ′(z4)∂C∂a4Δb1.(120)

Dividing by Δb1 we do indeed get the desired expression forthe gradient:

∂C∂b1=σ′(z1)w2σ′(z2)…σ′(z4)∂C∂a4.(121)

Why the vanishing gradient problem occurs: To understand whythe vanishing gradient problem occurs, let's explicitly write out theentire expression for the gradient:

∂C∂b1=σ′(z1)w2σ′(z2)w3σ′(z3)w4σ′(z4)∂C∂a4.(122)

Excepting the very last term, this expression is a product of terms ofthe form wjσ′(zj) . To understand how each of those termsbehave, let's look at a plot of the function σ′

:

-4-3-2-1012340.000.050.100.150.200.25zDerivative of sigmoid function

The derivative reaches a maximum at σ′(0)=1/4

. Now, if weuse our standard approachto initializing the weights in the network, then we'll choose theweights using a Gaussian with mean 0 and standard deviation 1 . Sothe weights will usually satisfy |wj|<1 . Putting theseobservations together, we see that the terms wjσ′(zj) willusually satisfy |wjσ′(zj)|<1/4

. And when we take aproduct of many such terms, the product will tend to exponentiallydecrease: the more terms, the smaller the product will be. This isstarting to smell like a possible explanation for the vanishinggradient problem.

To make this all a bit more explicit, let's compare the expression for ∂C/∂b1

to an expression for the gradient withrespect to a later bias, say ∂C/∂b3 . Of course,we haven't explicitly worked out an expression for ∂C/∂b3 , but it follows the same pattern described above for ∂C/∂b1

. Here's the comparison of the twoexpressions:

The two expressions share many terms. But the gradient ∂C/∂b1 includes two extra terms each of the form wjσ′(zj) . As we've seen, such terms are typically less than 1/4 in magnitude. And so the gradient ∂C/∂b1 willusually be a factor of 16 (or more) smaller than ∂C/∂b3 . This is the essential origin of the vanishinggradient problem.

Of course, this is an informal argument, not a rigorous proof that thevanishing gradient problem will occur. There are several possibleescape clauses. In particular, we might wonder whether the weights wj

could grow during training. If they do, it's possible the terms wjσ′(zj) in the product will no longer satisfy |wjσ′(zj)|<1/4 . Indeed, if the terms get large enough -greater than 1

- then we will no longer have a vanishing gradientproblem. Instead, the gradient will actually grow exponentially as wemove backward through the layers. Instead of a vanishing gradientproblem, we'll have an exploding gradient problem.

The exploding gradient problem: Let's look at an explicitexample where exploding gradients occur. The example is somewhatcontrived: I'm going to fix parameters in the network in just theright way to ensure we get an exploding gradient. But even though theexample is contrived, it has the virtue of firmly establishing thatexploding gradients aren't merely a hypothetical possibility, theyreally can happen.

There are two steps to getting an exploding gradient. First, wechoose all the weights in the network to be large, say w1=w2=w3=w4=100

. Second, we'll choose the biases so that the σ′(zj) terms are not too small. That's actually pretty easyto do: all we need do is choose the biases to ensure that the weightedinput to each neuron is zj=0 (and so σ′(zj)=1/4 ). So,for instance, we want z1=w1a0+b1=0 . We can achieve thisby setting b1=−100∗a0 . We can use the same idea to select theother biases. When we do this, we see that all the terms wjσ′(zj) are equal to 100∗14=25

. With thesechoices we get an exploding gradient.

The unstable gradient problem: The fundamental problem hereisn't so much the vanishing gradient problem or the exploding gradientproblem. It's that the gradient in early layers is the product ofterms from all the later layers. When there are many layers, that'san intrinsically unstable situation. The only way all layers canlearn at close to the same speed is if all those products of termscome close to balancing out. Without some mechanism or underlyingreason for that balancing to occur, it's highly unlikely to happensimply by chance. In short, the real problem here is that neuralnetworks suffer from an unstable gradient problem. As aresult, if we use standard gradient-based learning techniques,different layers in the network will tend to learn at wildly differentspeeds.

Exercise

• In our discussion of the vanishing gradient problem, we made use of the fact that |σ′(z)|<1/4

• . Suppose we used a different activation function, one whose derivative could be much larger. Would that help us avoid the unstable gradient problem?

The prevalence of the vanishing gradient problem: We've seenthat the gradient can either vanish or explode in the early layers ofa deep network. In fact, when using sigmoid neurons the gradient willusually vanish. To see why, consider again the expression |wσ′(z)|

. To avoid the vanishing gradient problem we need |wσ′(z)|≥1 . You might think this could happen easily if w is very large. However, it's more difficult than it looks. Thereason is that the σ′(z) term also depends on w : σ′(z)=σ′(wa+b) , where a is the input activation. So when we make w large, we need to be careful that we're not simultaneously making σ′(wa+b) small. That turns out to be a considerableconstraint. The reason is that when we make w large we tend to make wa+b very large. Looking at the graph of σ′ you can see thatthis puts us off in the "wings" of the σ′

function, where ittakes very small values. The only way to avoid this is if the inputactivation falls within a fairly narrow range of values (thisqualitative explanation is made quantitative in the first problembelow). Sometimes that will chance to happen. More often, though, itdoes not happen. And so in the generic case we have vanishinggradients.

Problems

• Consider the product |wσ′(wa+b)|

. Suppose |wσ′(wa+b)|≥1 . (1) Argue that this can only ever occur if |w|≥4 . (2) Supposing that |w|≥4 , consider the set of input activations a for which |wσ′(wa+b)|≥1 . Show that the set of a satisfying that constraint can range over an interval no greater in width than

2|w|ln(|w|(1+1−4/|w|−−−−−−−−√)2−1).(123)

(3) Show numerically that the above expression bounding the width of the range is greatest at |w|≈6.9 , where it takes a value ≈0.45

• . And so even given that everything lines up just perfectly, we still have a fairly narrow range of input activations which can avoid the vanishing gradient problem.
• Identity neuron: Consider a neuron with a single input, x , a corresponding weight, w1 , a bias b , and a weight w2 on the output. Show that by choosing the weights and bias appropriately, we can ensure w2σ(w1x+b)≈x for x∈[0,1] . Such a neuron can thus be used as a kind of identity neuron, that is, a neuron whose output is the same (up to rescaling by a weight factor) as its input. Hint: It helps to rewrite x=1/2+Δ , to assume w1 is small, and to use a Taylor series expansion in w1Δ
  • .

  Unstable gradients in more complex networks

  We've been studying toy networks, with just one neuron in each hiddenlayer. What about more complex deep networks, with many neurons ineach hidden layer?

  

  In fact, much the same behaviour occurs in such networks. In theearlier chapter on backpropagation we saw that the gradient in the l

  th layer of an L

  layer networkis given by:

  δl=Σ′(zl)(wl+1)TΣ′(zl+1)(wl+2)T…Σ′(zL)∇aC(124)

  Here, Σ′(zl)

  is a diagonal matrix whose entries are the σ′(z) values for the weighted inputs to the l th layer. The wl are the weight matrices for the different layers. And ∇aC is the vector of partial derivatives of C

  with respect to theoutput activations.

  This is a much more complicated expression than in the single-neuroncase. Still, if you look closely, the essential form is very similar,with lots of pairs of the form (wj)TΣ′(zj)

  . What's more,the matrices Σ′(zj) have small entries on the diagonal, nonelarger than 14 . Provided the weight matrices wj aren'ttoo large, each additional term (wj)TΣ′(zl)

  tends to makethe gradient vector smaller, leading to a vanishing gradient. Moregenerally, the large number of terms in the product tends to lead toan unstable gradient, just as in our earlier example. In practice,empirically it is typically found in sigmoid networks that gradientsvanish exponentially quickly in earlier layers. As a result, learningslows down in those layers. This slowdown isn't merely an accident oran inconvenience: it's a fundamental consequence of the approach we'retaking to learning.

  Other obstacles to deep learning

  In this chapter we've focused on vanishing gradients - and, moregenerally, unstable gradients - as an obstacle to deep learning. Infact, unstable gradients are just one obstacle to deep learning,albeit an important fundamental obstacle. Much ongoing research aimsto better understand the challenges that can occur when training deepnetworks. I won't comprehensively summarize that work here, but justwant to briefly mention a couple of papers, to give you the flavor ofsome of the questions people are asking.

  As a first example, in 2010 Glorot andBengio**Understanding the difficulty of training deep feedforward neural networks, by Xavier Glorot and Yoshua Bengio (2010). See also the earlier discussion of the use of sigmoids in Efficient BackProp, by Yann LeCun, Léon Bottou, Genevieve Orr and Klaus-Robert Müller (1998).found evidence suggesting that the use of sigmoid activation functionscan cause problems training deep networks. In particular, they foundevidence that the use of sigmoids will cause the activations in thefinal hidden layer to saturate near 0

  early in training,substantially slowing down learning. They suggested some alternativeactivation functions, which appear not to suffer as much from thissaturation problem.

  As a second example, in 2013 Sutskever, Martens, Dahl andHinton**On the importance of initialization and momentum in deep learning, by Ilya Sutskever, James Martens, George Dahl and Geoffrey Hinton (2013). studied the impact on deep learning of both the randomweight initialization and the momentum schedule in momentum-basedstochastic gradient descent. In both cases, making good choices madea substantial difference in the ability to train deep networks.

  These examples suggest that "What makes deep networks hard totrain?" is a complex question. In this chapter, we've focused on theinstabilities associated to gradient-based learning in deep networks.The results in the last two paragraphs suggest that there is also arole played by the choice of activation function, the way weights areinitialized, and even details of how learning by gradient descent isimplemented. And, of course, choice of network architecture and otherhyper-parameters is also important. Thus, many factors can play a rolein making deep networks hard to train, and understanding all thosefactors is still a subject of ongoing research. This all seems ratherdownbeat and pessimism-inducing. But the good news is that in thenext chapter we'll turn that around, and develop several approaches todeep learning that to some extent manage to overcome or route aroundall these challenges.