One hidden layer Neural Network - Random Initialization

When you train your NN, it's important to initialize the weights ( etc.) randomly. For logistic regression, it's ok to initialize the weights to 0; but for NN, if initialize the weights all to 0 and then apply gradient descent, it won't work!

The reason is that, if you initialize the weights W to zero, then all the hidden units are symmetric. And no matter how long you train your NN, all hidden units still computing exactly the same function. There is really no point to having more than one hidden unit because they're computing the same thing.

The solution is to initialize the weights randomly. Take following NN as an example:

figure-1

• Firstly, initialize  as following.

>>> W1=np.random.randn(2,2)*0.01
>>> W1
array([[-3.61903370e-03, -8.29109877e-05],
       [ 2.39660772e-03,  2.34993952e-03]])

• Initialize as all zero is fine

>>> b1=np.zeros((2,1))
>>> b1
array([[0.],
       [0.]])

• Similarly, you can initialize and :

>>> W2=np.random.randn(1,2)*0.01
>>> W2
array([[ 0.01702848, -0.00522487]])
>>> b2=np.zeros((1,1))
>>> b2
array([[0.]])
>>>

• Note we initialize  and  to very small random values by * 0.01. The reason is, if we're using sigmoid or tanh activation function, if the weights are too large or too small, then Z will be very big or very small. It's more likely that you'll end up with at the flat parts of the activation function where the slope of the gradient is very small. As a result, gradient descent will be very slow and learning will be very slow. If you don't have any tanh or sigmod function throughout you NN, this is less of an issue. But if you're doing binary classification, and your output unit is a sigmoid function, please don't initialize the weights too large.
• When you train a one hidden layer shallow NN, it's probably ok to use * 0.01. But when train a very deep NN, you'll probably pick a different constant. We'll discuss this later

<end>