Machine Learning Week 4

Neural Networks

Nonlinear Hypothesis

quadratic features($sum(x_i*x_j)=sum(x_i)/2$)
There will be too many features if we use the linear hypothesis when there has a lot of original features.

Neural Networks

To mimic human brain

Neuron Model: Logistic Unit

Single Neuron

Neural Network is a group of single neuron
Input layer - hidden layer - output layer (used to compute the hypothesis function)

Notations used in neural network

unit i of layer j

$$a_i^{(j)}$$
layer j-1, unit i in layer j, unit q in layer j-1, $$\Theta_{iq}^{(j-1)}$$
$$\Theta^{j} = S_{j+1}*(S_j+1)$$

Vectorised implementation

$$+a_0^{(j)}=1$$
$$z^{(j+1)}=\Theta^{(j)}a^{(j)}$$
$$a^{(j+1)}=g(z^{(j+1)})$$

Logistic Calculation Using Neural Network

And function

Or function

XNor Function

Multiple output units: One-vs-all

Weekly Matlab Exercise

%lrCostFunction

%theta2=theta(2:size(theta,1),1)
%J=1/m*(-y'*log(sigmoid(X*theta))-(1.-y)'*log(1-sigmoid(X*theta)))+lambda./(2*m)*(theta2'*theta2)
%grad1=(1/m*(sigmoid(X*theta)-y)'*X)'
%theta1=lambda/m*theta
%theta1(1,1)=0
%grad=grad1+theta1
theta2=theta(2:size(theta,1),1)'*theta(2:size(theta,1),1);
J=1/m*(-y'*log(sigmoid(X*theta))-(1.-y)'*log(1-sigmoid(X*theta)))+lambda/(2*m)*theta2;
grad1=1/m*X'*(sigmoid(X*theta)-y)+lambda/m*theta;
grad2=1/m*X'*(sigmoid(X*theta)-y);
grad2(2:end,1)=grad1(2:end,1);
grad=grad2;


%oneVsAll

options=optimset('GradObj','on','MaxIter',50);
for i=1:num_labels
initial_theta=zeros(n+1,1);
i_indices=find(y==i);
theta=fmincg(@(t)(lrCostFunction(t,X,y==i,lambda)),...
             initial_theta,options);
all_theta1(i,:)=theta';
end
all_theta=all_theta1


%predictOneVsAll

[values,indeces]=max(sigmoid(X*all_theta'),[],2)
p=indeces'


%predict

X=[ones(size(X,1),1) X];
A2=sigmoid(Theta1*X');
A2=[ones(1,size(A2,2));A2];
A3=sigmoid(Theta2*A2)
[vals,inds]=max(A3,[],1);
p=inds'