ML Notes: Week 4 - Neural Networks: Representation

1. Model representation

1.1 Neural network model

The typical neuron has input wires which called the dendrites and also has an output wire called an Axon. The nucleus is considered as the computational unit. We could simplify the model as follows:

Terms: a neuron or an artificial neuron with a sigmoid or logistic activation function

1.2 Some notations in the neural networks

layer 1: Input layer
layer 2: Hidden layer (the rest layers all could be called hidden layer)
layer 3: Output layer

• $x,\Theta$ are parameter vectors. In addition, the $\Theta$ also is called as weights.
• ${\Theta }_{in}^{(j)}$ = matrix of weights mapping from layer $j$ in layer $j+1$. If a network has $s_j$ units in layer $j$ and has $s_{j+1}$ units in layer $j+1$, then ${\Theta }^{(j)}=s_{j+1}\ast (s_j+1)$.
• $a_i^{(j)}$ = “activation” of unit $i$ in layer $j$.

$$\begin{gathered} \begin{array}{rl}{a}_1^{(2)}& =g({\Theta }_{10}^{(1)}{x}_0+{\Theta }_{11}^{(1)}{x}_1+{\Theta }_{12}^{(1)}{x}_2+{\Theta }_{13}^{(1)}{x}_3)\\ a_2^{(2)}& =g({\Theta }_{20}^{(1)}{x}_0+{\Theta }_{21}^{(1)}{x}_1+{\Theta }_{22}^{(1)}{x}_2+{\Theta }_{23}^{(1)}{x}_3)\\ a_3^{(2)}& =g({\Theta }_{30}^{(1)}{x}_0+{\Theta }_{31}^{(1)}{x}_1+{\Theta }_{32}^{(1)}{x}_2+{\Theta }_{33}^{(1)}{x}_3)\\ \\ {h}_{\Theta }(x)=a_1^{(3)}& =g({\Theta }_{10}^{(2)}{a}_0^{(2)}+{\Theta }_{11}^{(2)}{a}_1^{(2)}+{\Theta }_{12}^{(2)}{a}_2^{(2)}+{\Theta }_{13}^{(2)}{a}_3^{(2)}) \\ \end{array} \end{gathered}$$*$g$ is sigmoid/logistic activation function.

1.3 Forward propagation nueral network

The process of computing the activations, shown in the above figure, from the input then the hidden then the output layer, and that’s also called forward propagation

Now, we will vectorize the model. We difine
$$\begin{array}{rl}{z}_1^{(2)}& ={\Theta }_{10}^{(1)}{x}_0+{\Theta }_{11}^{(1)}{x}_1+{\Theta }_{12}^{(1)}{x}_2+{\Theta }_{13}^{(1)}{x}_3\\ z_2^{(2)}& ={\Theta }_{20}^{(1)}{x}_0+{\Theta }_{21}^{(1)}{x}_1+{\Theta }_{22}^{(1)}{x}_2+{\Theta }_{23}^{(1)}{x}_3\\ z_3^{(2)}& ={\Theta }_{30}^{(1)}{x}_0+{\Theta }_{31}^{(1)}{x}_1+{\Theta }_{32}^{(1)}{x}_2+{\Theta }_{33}^{(1)}{x}_3\end{array}$$
we can rewrite it as $z^{(2)}=[z_1^{(2)}{z}_1^{(2)}{z}_1^{(2)}{]}^T={\Theta }^{(1)}x$. If we treat $x$ as $a^{(1)}$, so $z^{(2)}={\Theta }^{(1)}{a}^{(1)}$ .
That is $z^{(j+1)}={\Theta }^{(j)}{a}^{(j)}$

And $a_1^{(2)}=g(z_1^{(2)}),a_2^{(2)}=g(z_2^{(2)}),a_3^{(2)}=g(z_3^{(2)})$ could be written as $a^{(2)}=g(z^{(2)})$ .

For the above neural network model, if we take the input layer away, the model just like logistic function.

Logistic function: $h_{\theta }(x)=g({\theta }_0+{\theta }_{1}x+{\theta }_2{x}_2)$
The simplified neural network model: $h_{\Theta }(x)=g({\Theta }_{10}^{(2)}{a}_0^{(2)}+{\Theta }_{11}^{(2)}{a}_1^{(2)}+{\Theta }_{12}^{(2)}{a}_2^{(2)}+{\Theta }_{13}^{(2)}{a}_3^{(2)})$

1.4 Other network architectures

2. How to compute a complex nonlinear function？

x 1 , x 2 ∈ { 0 , 1 } x_1,x_2 \in\{0,1\} x1​,x2​∈{0,1}

2.1 AND

$y=x_1$ AND $x_2$

${\Theta }^{(1)}=\left[\begin{array}{ccc}-30& 20& 20\end{array}\right]$

2.2 OR

$y=x_1$ OR $x_2$

${\Theta }^{(1)}=\left[\begin{array}{ccc}-10& 20& 20\end{array}\right]$

2.3 NOT

$y=$ NOT $x_1$

${\Theta }^{(1)}=\left[\begin{array}{cc}10& -20\end{array}\right]$

2.4 (NOT $x1$) AND (NOT $x_2$)

${\Theta }^{(1)}=\left[\begin{array}{ccc}10& -20& -20\end{array}\right]$

2.5 XNOR

$y=(x_1$ AND $x_2)$ OR $($(NOT $x1$) AND (NOT $x_2$)$)$

* we are able to put pieces together to generate some new functions.

3. Multi-class Classification

The ouput $y_i$ will be $\left[\begin{array}{c}1\\ 0\\ 0\\ 0\end{array}\right]$ ,$\left[\begin{array}{c}0\\ 1\\ 0\\ 0\end{array}\right]$ ,$\left[\begin{array}{c}0\\ 0\\ 1\\ 0\end{array}\right]$ ,$\left[\begin{array}{c}0\\ 0\\ 0\\ 1\end{array}\right]$ depending on what the corresponding input $X_i$ is. And in this way, we could implement the multi-class Classification.