ML Notes: Week 1 - Univariate Linear Regression

1. The Basic Theory

Hypothesis: $h_{\theta }(x)={\theta }_0+{\theta }_{1}x$

Parameters: ${\theta }_0,{\theta }_1$

Cost Function: $J(\theta )=\frac{1}{2m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)}{)}^2$

Objective Function: $\underset{{\theta }_0,{\theta }_1}{\min }J(\theta )$

2. Gradient descent of linear regression

2.1 What is the gradient descent ?

We adopt the Gradient Descent Method to find the optimum of ${\theta }_0,{\theta }_1$

${\theta }_j={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}J({\theta }_0,{\theta }_1)={\theta }_j-\frac{\alpha }{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})\frac{\partial h_{\theta }(x)}{\partial {\theta }_j}$

so, when $j=0$, ${\theta }_0={\theta }_0-\frac{\alpha }{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})$.

     $j=1$, ${\theta }_1={\theta }_1-\frac{\alpha }{m}\sum _{i=1}^m[(h_{\theta }(x^{(i)})-y^{(i)})\ast x^{(i)}]$.

We’d better to update the parameters simultaneously.

2.2 How to implement it？

STEP 1: Initialize the ${\theta }_0,{\theta }_1$

STEP 2: Calculate the $h_{\theta }(x)$

STEP 3: Come up with the new ${\theta }_0,{\theta }_1$ under the condition of the $h_{\theta }(x)$ in STEP 2

STEP 4: Loop from STEP 2 - STEP 3

When to stop? We can set the times of iteration.

2.3 Matlab code for gradient descent

%% ================== Data Generation ===================
x = normrnd(0, 0.6, 100, 1);
noise = normrnd(0, 0.03, 100, 1);
y = 0.43 + 0.22 * x + noise;

%% ================== Linear Regression ===================
m = length(y);
X = [ones(m,1),x];
% initialize the theta
theta = [0;0];theta_length = length(theta);
itera = 5000;
alpha = 0.01;
costFunc = zeros(m,1);
theta_itera = zeros(m,2);

for j = 1:itera
    theta_itera(j,:) = theta';  % record all the theta values during the process
    hypothesis = X * theta;
    costFunc(j) = (1/2*m) * sum(((hypothesis - y) .^ 2)); % cost function

    % using the gradienet descent algorithm to find the optimum
    for i = 1:theta_length
        theta(i) = theta(i) - (alpha/m) * ((hypothesis - y)'* X(:,i));  
    end
end

%% ================== Visualization ===================
%% plot the dataset and the fit line
figure;
for j = 1:10:itera
    plot(x,y,'bx');
    hold on
    
    h2 = plot(X(:,2),X*theta_itera(j,:)','r');
    pause(0.5);
    delete(h2);
end


%% plot the iterate curve
figure;
plot3(theta_itera(:,1),theta_itera(:,2),costFunc,'r');
xlabel('\theta_0'); ylabel('\theta_1');
hold on

%% plot the costFunction
% Grid over which we will calculate J
theta0_vals = linspace(-1, 1, 100);
theta1_vals = linspace(-1, 1, 100);
plot_costFunc = zeros(length(theta0_vals), length(theta1_vals));
% Fill 
for i = 1:length(theta0_vals)
    for j = 1:length(theta1_vals)
      t = [theta0_vals(i); theta1_vals(j)];
      plot_hypothesis = X * t;
      plot_costFunc(i,j) = (1/2*m) * sum(((plot_hypothesis - y) .^ 2));
    end
end

% Surface plot
%   In this case, the vertices of the surface patches are the triples
%   (x(j), y(i), Z(i,j)). Note that x corresponds to the columns of Z 
%   and y corresponds to the rows.
plot_costFunc = plot_costFunc';
surf(theta0_vals, theta1_vals,plot_costFunc)
xlabel('\theta_0'); ylabel('\theta_1');

%% Contour map of the costFunction
figure;
contour(theta0_vals, theta1_vals,  plot_costFunc, logspace(-2, 3, 20))
xlabel('\theta_0'); ylabel('\theta_1');
hold on;
plot(theta_itera(:,1), theta_itera(:,2), 'r');