ML Notes: Week 5 - Neural Networks: Learning

1. Cost function for neural networks

$\begin{array}{rl}J(\Theta )=& -\frac{1}{m}\sum _{i=1}^m\sum _{k=1}^K\left[y_k^{(i)}\log ((h_{\Theta }(x^{(i)}){)}_k)+(1-y_k^{(i)})\log (1-(h_{\Theta }(x^{(i)}){)}_k)\right]\\ & +\frac{\lambda }{2m}\sum _{l=1}^{L-1}\sum _{i=1}^{s_l}\sum _{j=1}^{s_{l+1}}({\Theta }_{j,i}^{(l)}{)}^2\end{array}$

• $m$ = number of samples
• $K$ = number of output units
• $(h_{\Theta }(x^{(i)}){)}_k$ = hypothesis that results in the $k^{th}$ output for $i^{th}$ sample
• $\lambda$ = regularization parameter
• $L$ = total number of layers in the network
• $s_l$ number of units (excluding the bias unit) in layer $l$.

Cost function for logistic regression:
$J(\theta )=-\frac{1}{m}\sum _{i=1}^m{y}^{(i)}\log h_{\theta }(x^{(i)})+(1-y^{(i)})\log (1-h_{\theta }(x^{(i)}))+\frac{\lambda }{2m}\sum _{j=1}^n{\theta }_j^2$

2. Understanding the backpropagation

In fitting a neural network, backpropagation computes the gradient of the loss function with respect to the weights of the network for a single input–output example, and does so efficiently, unlike a naive direct computation of the gradient with respect to each weight individually¹. In other words, backpropagation could help us minimize our cost function fo neural networks.

For the given training set { ( x ( 1 ) , y ( 1 ) ) ⋯ ( x ( m ) , y ( m ) ) } \lbrace (x^{(1)}, y^{(1)}) \cdots (x^{(m)}, y^{(m)})\rbrace {(x(1),y(1))⋯(x(m),y(m))}, the backpropagation algorithm will be implemented with the following steps:

• Obtaining the output value(activation) of the output layer. If the network have $L$ layers, we could calculated the $a^{(L)}$ by forward propagation algorithm.

• Computing error term for the ouput layer ${\delta }_k^{(L)}=a_k^{(L)}-y_k$

• Computing error term for hidden layers ${\delta }^{(L-1)},{\delta }^{(L-2)},\dots ,{\delta }^{(2)}$
  $$\begin{array}{rl}{\delta }^{(l)}& =({\Theta }^{(l)}{)}^T{\delta }^{(l+1)}.\ast g^{\prime }(z^{(l)})\\ & =({\Theta }^{(l)}{)}^T{\delta }^{(l+1)}.\ast g(z^{(l)}).\ast (1-g(z^{(l)}))\\ & =({\Theta }^{(l)}{)}^T{\delta }^{(l+1)}.\ast a^{(l)}.\ast (1-a^{(l)})\end{array}$$

if $g(z)=\frac{1}{1+e^{-z}}$,$g^{\prime }(z)=$ ?

$$\begin{array}{rl}{g}^{\prime }(z)& =-(\frac{1}{1+e^{-z}}{)}^2\cdot e^{-z}\cdot (-1)\\ & =\frac{1+e^{-z}-1}{(1+e^{-z})(1+e^{-z})}\\ & =\frac{1+e^{-z}}{(1+e^{-z})(1+e^{-z})}-\frac{1}{(1+e^{-z})(1+e^{-z})}\\ & =\frac{1}{1+e^{-z}}-\frac{1}{(1+e^{-z})(1+e^{-z})}\\ & =\frac{1}{1+e^{-z}}\cdot \left(1-\frac{1}{1+e^{-z}}\right)\\ & =g(z)\cdot (1-g(z))\end{array}$$

• Accumulating the gradient using ${\Delta }^{(l)}:={\Delta }^{(l)}+{\delta }^{(l+1)}(a^{(l)}{)}^T$
  * here the weights for bias unit should be removed.
  * The formula also can be rewritten as ${\Delta }^{(l)}=\sum _{i=1}^m{\left({\delta }^{(l+1)}\right)}^i{\left((a^{(l)}{)}^T\right)}^i$

• Obtaining the gradient for the neural network
  $$\begin{array}{rlr}\frac{\partial }{\partial {\Theta }_{ij}^{(l)}}J(\Theta )& =D_{ij}^{(l)}=\frac{1}{m}{\Delta }_{ij}^{(l)}& forj=0\\ \frac{\partial }{\partial {\Theta }_{ij}^{(l)}}J(\Theta )& =D_{ij}^{(l)}=\frac{1}{m}{\Delta }_{ij}^{(l)}+\frac{\lambda }{m}{\Theta }_{ij}^{(l)}& forj\ge 1\end{array}$$

Now, a simple neural network² is given to illustrate the back propagation algorithm.


(1) Firstly, we generate a 3-layer neural network with teo inputs, two hidden neurons and two output neurons, and intialize the weights as follows.

For hidden layers,
$$z_1^{(2)}=0.35\ast a_0^{(1)}+w_1\ast a_1^{(1)}+w_2\ast a_2^{(1)}$$
so the activiation of neuron $a_1^{(2)}$
$$a_1^{(2)}=\frac{1}{1+e^{-z_1^{(2)}}}$$
Carrying out the same process we could get $a_2^{(2)}$
Then, we repeat the process for the output layer neurons, using the output $a^{(2)}$ as inputs.
$$z_1^{(3)}=0.60\ast a_0^{(2)}+w_5\ast a_1^{(2)}+w_6\ast a_2^{(2)}$$
$$a_1^{(3)}=\frac{1}{1+e^{-z_1^{(3)}}}$$
Here, we difine the total error for output neuron summing the squared error:
$$E_{total}=\sum _{k=1}^K\frac{1}{2}(y_k-a_k^{(3)}{)}^2$$
For the first output neuron, its error
$$E_{a_1^{(3)}}=\frac{1}{2}(y_1-a_1^{(3)}{)}^2$$
and the error of the second output neuron
$$E_{a_2^{(3)}}=\frac{1}{2}(y_2-a_2^{(3)}{)}^2$$


(2)The backwards pass
By applying the chain rule we know that:

$$\frac{\partial E_{total}}{\partial w_5}=\underset{{\delta }_1^{(3)}}{\underbrace{\frac{\partial E_{total}}{\partial a_1^{(3)}}\ast \frac{\partial a_1^{(3)}}{\partial z_1^{(3)}}}}\ast \frac{\partial z_1^{(3)}}{\partial w_5}$$
Now, we need to figure out these partial derivatives.

• (1) $E_{total}=E_{a_1^{(3)}}+E_{a_2^{(3)}}=\frac{1}{2}(y_1-a_1^{(3)}{)}^2+\frac{1}{2}(y_2-a_2^{(3)}{)}^2$
   $\frac{\partial E_{total}}{\partial a_1^{(3)}}=2\ast \frac{1}{2}(y_1-a_1^{(3)})\ast (-1)+0=-(y_1-a_1^{(3)})$
• (2) $a_1^{(3)}=\frac{1}{1+e^{-z_1^{(3)}}}=g(z_1^{(3)})$
   $\frac{\partial a_1^{(3)}}{\partial z_1^{(3)}}=g^{\prime }(z_1^{(3)})=g(z_1^{(3)})\ast (1-g(z_1^{(3)}))=a_1^{(3)}\ast (1-a_1^{(3)})$
• (3) $z_1^{(3)}=0.60\ast a_0^{(2)}+w_5\ast a_1^{(2)}+w_6\ast a_2^{(2)}$
    $\frac{\partial z_1^{(3)}}{\partial w_5}=a_1^{(2)}$
  $$\frac{\partial E_{total}}{\partial w_5}=\underset{{\delta }_1^{(3)}}{\underbrace{-(y_1-a_1^{(3)})\ast a_1^{(3)}\ast (1-a_1^{(3)})}}\ast a_1^{(2)}$$

The above formula also can be represented as ${\Delta }_1^{(3)}={\delta }_1^{(3)}\ast a_1^{(2)}$
So, we get the gradient with respect to $w_5$ and also can get the new weights of $w_6,w_7$ and $w_8$ by repeating the above process.


Next, we’ll continue the backwards pass by calculating new values for $w_1,w_2,w_3$ and $w_4$.

$$\begin{array}{rl}\frac{\partial E_{total}}{\partial w_1}=& \underset{\Delta node1}{\underbrace{\frac{\partial E_{total}}{\partial a_1^{(3)}}\ast \frac{\partial a_1^{(3)}}{\partial z_1^{(3)}}\ast \frac{\partial z_1^{(3)}}{\partial a_1^{(2)}}\ast \frac{\partial a_1^{(2)}}{\partial z_1^{(2)}}\ast \frac{\partial z_1^{(2)}}{\partial w_1}}}+\\ & \underset{\Delta node2}{\underbrace{\frac{\partial E_{total}}{\partial a_2^{(3)}}\ast \frac{\partial a_2^{(3)}}{\partial z_2^{(3)}}\ast \frac{\partial z_2^{(3)}}{\partial a_1^{(2)}}\ast \frac{\partial a_1^{(2)}}{\partial z_1^{(2)}}\ast \frac{\partial z_1^{(2)}}{\partial w_1}}}\end{array}$$

from the $\frac{\partial E_{total}}{\partial w_5}$ we get $\frac{\partial E_{total}}{\partial a_1^{(3)}}\ast \frac{\partial a_1^{(3)}}{\partial z_1^{(3)}}={\delta }_1^{(3)}$

• (1) $z_1^{(3)}=0.60\ast a_0^{(2)}+w_5\ast a_1^{(2)}+w_6\ast a_2^{(2)}$
    $\frac{\partial z_1^{(3)}}{\partial a_1^{(2)}}=w^5$
• (2) $a_1^{(2)}=\frac{1}{1+e^{-z_1^{(2)}}}=g(z_1^{(2)})$
   $\frac{\partial a_1^{(2)}}{\partial z_1^{(2)}}=g^{\prime }(z_1^{(2)})=g(z_1^{(2)})\ast (1-g(z_1^{(2)}))=a_1^{(2)}\ast (1-a_1^{(2)})$
• (3) $z_1^{(2)}=0.35\ast a_0^{(2)}+w_1\ast a_1^{(1)}+w_2\ast a_2^{(1)}$
    $\frac{\partial z_1^{(2)}}{\partial w_1}=a_1^{(1)}$
  
  

so, $\Delta node1=\underset{{\delta }_{11}^{(2)}}{\underbrace{{\delta }_1^{(3)}\ast w^5\ast g^{\prime }(z_1^{(2)})}}\ast a_1^{(1)}$, and $\Delta node2=\underset{{\delta }_{12}^{(2)}}{\underbrace{{\delta }_2^{(3)}\ast w^7\ast g^{\prime }(z_1^{(2)})}}\ast a_1^{(1)}$
$$\frac{\partial E_{total}}{\partial w_1}=\Delta node1+\Delta node2$$
We rewrite the gradient as ${\Delta }_1^{(2)}=\Delta node1+\Delta node2={\delta }_{11}^{(2)}\ast a_1^{(1)}+{\delta }_{12}^{(2)}\ast a_1^{(1)}={\delta }_1^{(2)}\ast a_1^{(1)}$

References

[1] Backpropagation

[2] A Step by Step Backpropagation Example (EN) - Matt Mazur
[3] A Step by Step Backpropagation Example (CN)

3. Gradient checking

Gradient checking will help to confirm that the backpropagation works correctly. We can approximate the derivative of our cost function with:
$$\frac{\partial }{\partial \Theta }J(\Theta )\approx \frac{J(\Theta +ϵ)-J(\Theta -ϵ)}{2ϵ}$$
Before performing the gradient checking, we unroll the parameters(theta) into a long vector $\theta$. Generaly, we set $ϵ=10^{-4}$ to guarantee the gradient.

4. Application of neural network to classification task

(1) Weights initialization

Initializing all theta weights to zero does not work with neural networks. Hence, we initialize all theta between $[-ϵ,ϵ]$ using the folowing method.

• *${ϵ}_{initial}=0.12$

W = zeros(L_out, 1 + L_in);
epsilon_init = 0.12;
W = rand(L_out, 1 + L_in) * 2 * epsilon_init - epsilon_init;

(2) Feedforward the neural network

% performing the forward propagation
	X = [ones(m,1),X];                % 5000*401
	h_out = sigmoid(X * Theta1');     % 5000*25
	h_out = [ones(m,1),h_out];        % 5000*26
	hypo = sigmoid(h_out * Theta2');  % 5000*10

% generating the label matrix
	y_label = zeros(m, num_labels);  
	for i = 1:num_labels
	    loc = find(y == i);
	    y_label(loc,i) = ones(size(loc,1),1);
	end

(3) Cost function computation

$$\begin{array}{rl}J(\Theta )=& -\frac{1}{m}\sum _{i=1}^m\sum _{k=1}^K\left[y_k^{(i)}\log ((h_{\Theta }(x^{(i)}){)}_k)+(1-y_k^{(i)})\log (1-(h_{\Theta }(x^{(i)}){)}_k)\right]\\ & +\frac{\lambda }{2m}\sum _{l=1}^{L-1}\sum _{i=1}^{s_l}\sum _{j=1}^{s_{l+1}}({\Theta }_{j,i}^{(l)}{)}^2\end{array}$$

% cost function (no regularization)
    J= (y_label.*log(hypo)) + ((ones(m,num_labels)-y_label).*log(1-hypo));
    J = sum(sum(J));
    J = (-1/m) * J;
    
% regularization term of the cost function
	theta_sum = 0;    
	Theta = []
	for i = 1:2
	    Theta = eval(['Theta',num2str(i)]);
	    theta_sum = theta_sum + sum(sum(Theta(:,2:end).^2));
	    clear Theta
	end
J = J + lambda / (2 * m) * theta_sum;

(4) Backpropagation (Gradient calculation)

$$\begin{array}{rl}{\delta }_k^{(L)}& =a_k^{(L)}-y_k\\ {\delta }^{(l)}& =({\Theta }^{(l)}{)}^T{\delta }^{(l+1)}.\ast a^{(l)}.\ast (1-a^{(l)})\end{array}$$

delta_3 = (hypo - y_label)';
delta_2 = Theta2(:,2:end)' * delta_3.*sigmoidGradient((X * Theta1')');

$${\Delta }^{(l)}:={\Delta }^{(l)}+{\delta }^{(l+1)}(a^{(l)}{)}^T$$
* Vectorization the above equation
$${\Delta }^{(l)}={\delta }^{(l+1)}{a}^{(l)}$$

delta_sum_1 = zeros(hidden_layer_size,input_layer_size+1);  % 25*401
delta_sum_2 = zeros(num_labels,hidden_layer_size+1);        % 10*26

delta_sum_1 = delta_2 * X;
delta_sum_2 = delta_3 * h_out;

$$\begin{array}{rlr}\frac{\partial }{\partial {\Theta }_{ij}^{(l)}}J(\Theta )& =D_{ij}^{(l)}=\frac{1}{m}{\Delta }_{ij}^{(l)}& forj=0\\ \frac{\partial }{\partial {\Theta }_{ij}^{(l)}}J(\Theta )& =D_{ij}^{(l)}=\frac{1}{m}{\Delta }_{ij}^{(l)}+\frac{\lambda }{m}{\Theta }_{ij}^{(l)}& forj\ge 1\end{array}$$

Theta1_grad = (1/m) .* delta_sum_1;
Theta2_grad = (1/m) .* delta_sum_2;

regular_1 = Theta1 * (lambda/m);
regular_1(:,1) = 0;
Theta1_grad = Theta1_grad + regular_1;

regular_2 = Theta2 * (lambda/m);
regular_2(:,1) = 0;;
Theta2_grad = Theta2_grad + regular_2;

grad = [Theta1_grad(:); Theta2_grad(:)];

(5) Gradients checking (optional)

$$\frac{\partial }{\partial {\Theta }_j}J(\Theta )\approx \frac{J({\Theta }_1,\cdots ,{\Theta }_j+ϵ,\cdots ,{\Theta }_n)-J({\Theta }_1,\cdots ,{\Theta }_j-ϵ,\cdots ,{\Theta }_n)}{2ϵ}$$

• *$ϵ=10^{-4}$

theta = [Theta1(:); Theta2(:)];
numgrad = zeros(size(theta));
perturb = zeros(size(theta));
e = 1e-4;
for p = 1:numel(theta)
    % Set perturbation vector
    perturb(p) = e;
    loss1 = J(theta - perturb);
    loss2 = J(theta + perturb);
    % Compute Numerical Gradient
    numgrad(p) = (loss2 - loss1) / (2*e);
    perturb(p) = 0;
end

*After checking with correct gradient, we could turn off gradient checking before running algorithm.

(6) Minimizing the cost function $J(\Theta )$

options = optimset('MaxIter', 200);
lambda = 0.2;

% Create "short hand" for the cost function to be minimized
costFunction = @(p) nnCostFunction(p, ...
                                   input_layer_size, ...
                                   hidden_layer_size, ...
                                   num_labels, X, y, lambda);

[nn_params, cost] = fmincg(costFunction, initial_nn_params, options);