Course1-week3-one hidden layer neural network

3.1 - neural networks overview

Some new notation have been introduce, we’ll use superscript square bracket 1 to refer to the layer of neural network, for instance, $w^{[1]}$ representing the parameters of layer 1, and this superscript square bracker we have here are not to confuse with the superscript round bracket which we used to refer to individual training example. so $x^{(i)}$ refer to the $i^{th}$ training example.

3.2 - neural network representation

3.3 - computing a neural network output

Let’s go more deeply into exactly what this neural network compute.

$$z^{[1]}_{1} = w^{[1]T}_{1}x + b^{[1]}_{1}, a^{[1]}_{1} = \sigma(z^{[1]}_{1}) \\ z^{[1]}_{2} = w^{[1]T}_{2}x + b^{[1]}_{2}, a^{[1]}_{2} = \sigma(z^{[1]}_{2}) \\ z^{[1]}_{3} = w^{[1]T}_{3}x + b^{[1]}_{3}, a^{[1]}_{3} = \sigma(z^{[1]}_{3}) \\ z^{[1]}_{4} = w^{[1]T}_{4}x + b^{[1]}_{4}, a^{[1]}_{4} = \sigma(z^{[1]}_{4}) \\ $$
take these four equations and vectorize.

$$\begin{equation} z^{[1]}=W^{[1]}x+b{[1]}= \left[ \begin{aligned} \cdots w^{[1]T}_{1} \cdots \\ \cdots w^{[1]T}_{2} \cdots \\ \cdots w^{[1]T}_{3} \cdots \\ \cdots w^{[1]T}_{4} \cdots \\ \end{aligned} \right] \left[ \begin{aligned} x_1 \\ x_2 \\ x_3 \\ \end{aligned} \right]+ \left[ \begin{aligned} b^{[1]}_{1} \\ b^{[1]}_{2} \\ b^{[1]}_{3} \\ b^{[1]}_{4} \\ \end{aligned} \right]= \left[ \begin{aligned} w^{[1]T}_{1}x + b^{[1]}_{1} \\ w^{[1]T}_{2}x + b^{[1]}_{2} \\ w^{[1]T}_{3}x + b^{[1]}_{3} \\ w^{[1]T}_{4}x + b^{[1]}_{4} \\ \end{aligned} \right]= \left[ \begin{aligned} z^{[1]}_1 \\ z^{[1]}_2 \\ z^{[1]}_3 \\ z^{[1]}_4 \\ \end{aligned} \right] \end{equation}$$

When we are vectorizing one of the rule of thumb that when we have different nodes in a layer, we stack them vertically.

$$z^{[1]} = \sigma(a^{[1]})$$

So when you have a neural network with one hidden layer, what you need to implement to compute the output is just the four equations below.

$$z^{[1]} = W^{[1]}x + b^{[1]} \\ a^{[1]} = \sigma(z^{[1]}) \\ z^{[2]} = W^{[2]}x + b^{[2]} \\ a^{[2]} = \sigma(z^{[2]}) \\ $$

Now we have seen how given a single input feature vector $x$ you can with four line of codes compute the output of this neural network. Similiar to what we did for logistic regression, we also want to vectorize across multiple training examples. So be able to compute the output of the neural network not just one example at a time, but your entire training set at a time.

3.4 - vectorizing across multiple examples

Stacking up different training examples in different columns of a matrix, we’ll able to take the four equations from the previous talk and very little modifiation.
If you have a unvectorized implementation to compute the output for all training examples, you need to for $i$ equal 1 to m.

$$x^{(1)} \longrightarrow a^{[2](1)} \\ x^{(2)} \longrightarrow a^{[2](2)} \\ \cdots \\ x^{(3)} \longrightarrow a^{[2](m)}$$

And the vectorized version is:

$$Z^{[1]} = W^{[1]}A^{[0]} + b^{[1]} \\ A^{[1]} = \sigma(Z^{[1]}) \\ Z^{[2]} = W^{[2]}A^{[1]} + b^{[2]} \\ A^{[2]} = \sigma(Z^{[2]}) \\ $$
where:
$$\begin{equation} X = A^{[0]} = \left[ \begin{aligned} & & & & \\ & x^{(1)} & x^{(2)} & \cdots & x^{(m)} \\ & & & & \\ \end{aligned} \right]_{(n_x, m)} \end{equation}$$

$$\begin{equation} Z^{[1]} = \left[ \begin{aligned} & & & & \\ & z^{[1](1)} & z^{[1](2)} & \cdots & z^{[1](m)} \\ & & & & \\ \end{aligned} \right] \end{equation}$$

$$\begin{equation} A^{[1]} = \left[ \begin{aligned} & & & & \\ & a^{[1](1)} & a^{[1](2)} & \cdots & a^{[1](m)} \\ & & & & \\ \end{aligned} \right] \end{equation}$$

So the horizontal, the matrix $A, Z$ goes over different training examples; vertical, the matrix goes over different hidden unit neural.

3.5 - explanation for vectorized implementation

3.6 - activation functions

$$sigmoid(z) = \frac{1}{1+e^{(-z)}}$$
$$tanh(z) = \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}}$$
It’s turn out if you take the activation function be equal to $tanh(z)$, this almost always better than the $sigmoid$ function, because the value between -1 and 1, the means of activation value that come out from your hidden layer are close to 0, this actually meke your learning for the next layer a little bit easier. I pertty much never use the $sigmoid$ activation function anymore, but the one exception is for the output layer, because if y is either 0 or 1, than you want to output just between 0 and 1 rather than between -1 and 1.

One of the downsides of both the sigmoid function and tanh function is that is z is very small or very large, then the gradient, the derivative or the slope of the function become very small, this can slow down the gradient descent.

$$Relu(z) = max(0, z)$$
$$leaky\ Rulu(z) = max(0.01z, z)$$

The advantage of Rule and leaky Relu is for a lot of space of z, the slope of the activation function is very different from 0, so in partice, using the Relu activation function your neural network will often learn much faster than using the tanh and sigmoid activation function.

3.7 - why do you need non-linear activation function

Why does the neural network need a nonlinear activation function? if you do this, your model is just computing $\hat{y}$ as a linear function of your input feature x.

$$\begin{aligned} a^{[1]} & = w^{[1]}x + b^{[1]} \\ a^{[2]} & = w^{[2]}a^{[1]} + b^{[2]} \\ \end{aligned}$$
so
$$\begin{aligned} a^{[2]} & = w^{[2]}(w^{[1]}x + b^{[1]}) + b^{[2]} \\ & = \underbrace{w^{[2]}w^{[1]}}x + \underbrace{w^{[2]}b^{[1]} + b^{[2]}} \\ & = w'x+b' \end{aligned}$$
just outputing a linear function of input x. One place you may use linear activation function is where the problem you have face is regression.

3.8 - derivatives of activation function

$$a = g(z) = \frac{1}{1 + e^{-z}}$$
$$g'(z) = a(1 - a)\tag1$$
$$a = g(z) = \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}}$$
$$g'(z) = 1 - a^2\tag2$$
$$g(z) = max(0, z)$$
$$\begin{equation} g'(z) = \left\{ \begin{aligned} 1, \ if \ z \ge 0; \\ 0, \ if \ z < 0; \end{aligned} \right.\tag3 \end{equation}$$

$$g(z) = max(0.001z, z)$$
$$\begin{equation} g'(z) = \left\{ \begin{aligned} 1, \ if \ z \ge 0; \\ 0.001, \ if \ z < 0; \end{aligned} \right.\tag4 \end{equation}$$

3.9 - gradient descent for neural networks

We will see how the implement gradient descent for neural network with one hidden layer. The parameters of neural network with one hidden layer are:

$$W^{[1]}_{(n^{[1]},n^{(0)})}, b^{[1]}_{(n^{[1]}, 1)}, W^{[2]}_{(n^{[2]},n^{(1)})}, b^{[2]}_{(n^{[2]}, (n^{[1]})}$$

assuming that we are doing binary classification, so the cost function is:

$$J(W^{[1]}, b^{[1]}, W^{[2]}, b^{[2]}) = \frac1m \sum_{i = 1}^{m} \mathcal{L}(\hat{h}, a^{[2]})$$

if you doing binary classification, the loss function can be exactly what you ues for logistic regression.

Let’s summarize again the equations for forward propagation:

$$Z^{[1]} = W^{[1]}X + b^{[1]} \\ A^{[1]} = g^{[1]}(Z^{[1]}) \\ Z^{(2)} = W^{[2]}A^{[1]} + b^{[2]} \\ A^{[2]} = g^{[2]}(Z^{[2]}) \\ $$

back propagation

$$\begin{aligned} & dZ^{[2]} = A^{[2]} - Y, Y = [y^{[1]}, y^{[2]}, \cdots, y^{[m]}], A = [a^{[1]}, a^{[2]}, \cdots, a^{[m]}] \\ & dW^{[2]} = \frac1m dZ^{[2]}A^{[1]T} \\ & db^{[2]} = \frac1m np.sum(dZ^{[2]}, axis = 1, keepdims = True) \\ & dZ^{[1]} = W^{[2]T}dZ^{[2]} * g^{[1]'}(Z^{[1]}), (g^{[1]'}(Z^{[1]}) \ is \ (n^{[1]}, m) \ and \ W^{[2]T}dZ^{[2]} \ is \ (n^{[1]}, m)) \\ & dW^{[1]} = \frac1m dZ^{[1]}X^T \\ & db^{[1]} = \frac1m np.sum(dZ^{[1]}, axis = 1, keepdims = True) \end{aligned}$$

3.10 - backpropagation intuition

backpropagation for logistic regression

backpropagation for neural network


equations for backpropagation

3-11 random initialization

When you training neural network, it’s important to initialize the weights randomly, for logisitic regression, it was okay to initialize the weights to zero, but for neural network to initialize the weights to all zero and apply the gradient descent it won’t work.

if you initialize the neural network:

$$W^{[1]} = \begin{bmatrix} 0 & 0 \\ 0 & 0 \\ \end{bmatrix} , W^{[2]} = [0 \ 0]$$
the two hidden unit are completely identical, so they compute the exactly same function, but we want different unit to compute the different things.

W1 = np.random.randn((n_1, n_0)) * 0.01
b = np.zeros((n_1, 0))

where did this constant 0.01 comes from, why is it 0.01, why not put the number 100 or 1000, it turns out that our usually prefer to initialize the paramenters very small random values. Because if you use a sigmoid or tanh activation function, if the paramenter are too large when you compute the activation values – $z^{(1)} = W^{[1]}x + b^{[1]}, a^{[1]} = g(z^{[1]})$, so if $W$ is very big, the $z$ will be very big, you will starting off training with very large of z z <script type="math/tex" id="MathJax-Element-42">z</script>, which causes tanh and sigmoid activation function to be saturated, thus slowing down learning.