(读书笔记）Unstable gradients in deep neural nets - Vanishing and Exploding

读书笔记 ： Michael Nielsen - Neural Networks and Deep Learning Chapter 5

1. The phenomenon of unstable gradients in deep neural nets
   For example : in a [784, 30, 30, 30, 10] network
   We use batch gradient descent with just 1000 training data set
   learning rate = 0.1, regularization parameter = 5.0, epochs = 500
   The gradient tends to get smaller as we move backward through the hidden layers, this is called vanishing of gradients problem as below chart.
   Sometimes the gradient gets much larger in earlier layers! This is the exploding gradient problem
   

2. Math explanation of the phenomenon
   2.1 Vanishing gradient problem
   Investigate with below simple example, just one single neuron in each layer. The activity function is just sigmoid function.
   

Using back propagation to update the bias, then the gradient function of b1 is :

To recap:
σ(z) = 1/(1+exp(-z))

The chart of σ’(z) is :

So the maximum value of the function is : σ′(0)=1/4
Revisit the function :
With |wj|<1, the product will tend to exponentially decrease: the more terms, the smaller the product will be.
This is the essential cause of vanishing gradient problem.
2.2 Exploding gradient problem
We can manually setup w1=w2=w3=w4=100
And choose bias to make zj=0,then σ′(zj)=1/4 (max value)
Then with below function of updating we can get a exploding gradient problem.