1. The Basic Theory
Hypothesis: h θ ( x ) = θ 0 + θ 1 x h_\theta(x)=\theta_0+\theta_1x hθ(x)=θ0+θ1x
Parameters: θ 0 , θ 1 \theta_0, \theta_1 θ0,θ1
Cost Function: J ( θ ) = 1 2 m ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) 2 J(\theta)=\frac{1}{2m}\sum\limits_{i=1}^m(h_\theta(x^{(i)})-y^{(i)})^2 J(θ)=2m1i=1∑m(hθ(x(i))−y(i))2
Objective Function: min θ 0 , θ 1 J ( θ ) \min\limits_{\theta_0,\theta_1}J(\theta) θ0,θ1minJ(θ)
2. Gradient descent of linear regression
2.1 What is the gradient descent ?
We adopt the Gradient Descent Method to find the optimum of θ 0 , θ 1 \theta_0,\theta_1 θ0,θ1
θ j = θ j − α ∂ ∂ θ j J ( θ 0 , θ 1 ) = θ j − α m ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) ∂ h θ ( x ) ∂ θ j \theta_j=\theta_j-\alpha \frac{\partial}{\partial\theta_j}J(\theta_0,\theta_1) =\theta_j-\frac{\alpha}{m}\sum\limits_{i=1}^m(h_\theta(x^{(i)})-y^{(i)})\frac{\partial h_\theta(x)}{\partial\theta_j} θj=θj−α∂θj∂J(θ0,θ1)=θj−mαi=1∑m(hθ(x(i))−y(i))∂θj∂hθ(x)
so, when j = 0 j=0 j=0, θ 0 = θ 0 − α m ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) \theta_0=\theta_0-\frac{\alpha}{m}\sum\limits_{i=1}^m(h_\theta(x^{(i)})-y^{(i)}) θ0=θ0−mαi=1∑m(hθ(x(i))−y(i)).
j = 1 j=1 j=1, θ 1 = θ 1 − α m ∑ i = 1 m [ ( h θ ( x ( i ) ) − y ( i ) ) ∗ x ( i ) ] \theta_1=\theta_1-\frac{\alpha}{m}\sum\limits_{i=1}^m[(h_\theta(x^{(i)})-y^{(i)})*x^{(i)}] θ1=θ1−mαi=1∑m[(hθ(x(i))−y(i))∗x(i)].
We’d better to update the parameters simultaneously.
2.2 How to implement it?
STEP 1: Initialize the θ 0 , θ 1 \theta_0,\theta_1 θ0,θ1
STEP 2: Calculate the h θ ( x ) h_\theta(x) hθ(x)
STEP 3: Come up with the new θ 0 , θ 1 \theta_0,\theta_1 θ0,θ1 under the condition of the h θ ( x ) h_\theta(x) hθ(x) in STEP 2
STEP 4: Loop from STEP 2 - STEP 3
When to stop? We can set the times of iteration.
2.3 Matlab code for gradient descent
%% ================== Data Generation ===================
x = normrnd(0, 0.6, 100, 1);
noise = normrnd(0, 0.03, 100, 1);
y = 0.43 + 0.22 * x + noise;
%% ================== Linear Regression ===================
m = length(y);
X = [ones(m,1),x];
% initialize the theta
theta = [0;0];theta_length = length(theta);
itera = 5000;
alpha = 0.01;
costFunc = zeros(m,1);
theta_itera = zeros(m,2);
for j = 1:itera
theta_itera(j,:) = theta'; % record all the theta values during the process
hypothesis = X * theta;
costFunc(j) = (1/2*m) * sum(((hypothesis - y) .^ 2)); % cost function
% using the gradienet descent algorithm to find the optimum
for i = 1:theta_length
theta(i) = theta(i) - (alpha/m) * ((hypothesis - y)'* X(:,i));
end
end
%% ================== Visualization ===================
%% plot the dataset and the fit line
figure;
for j = 1:10:itera
plot(x,y,'bx');
hold on
h2 = plot(X(:,2),X*theta_itera(j,:)','r');
pause(0.5);
delete(h2);
end
%% plot the iterate curve
figure;
plot3(theta_itera(:,1),theta_itera(:,2),costFunc,'r');
xlabel('\theta_0'); ylabel('\theta_1');
hold on
%% plot the costFunction
% Grid over which we will calculate J
theta0_vals = linspace(-1, 1, 100);
theta1_vals = linspace(-1, 1, 100);
plot_costFunc = zeros(length(theta0_vals), length(theta1_vals));
% Fill
for i = 1:length(theta0_vals)
for j = 1:length(theta1_vals)
t = [theta0_vals(i); theta1_vals(j)];
plot_hypothesis = X * t;
plot_costFunc(i,j) = (1/2*m) * sum(((plot_hypothesis - y) .^ 2));
end
end
% Surface plot
% In this case, the vertices of the surface patches are the triples
% (x(j), y(i), Z(i,j)). Note that x corresponds to the columns of Z
% and y corresponds to the rows.
plot_costFunc = plot_costFunc';
surf(theta0_vals, theta1_vals,plot_costFunc)
xlabel('\theta_0'); ylabel('\theta_1');
%% Contour map of the costFunction
figure;
contour(theta0_vals, theta1_vals, plot_costFunc, logspace(-2, 3, 20))
xlabel('\theta_0'); ylabel('\theta_1');
hold on;
plot(theta_itera(:,1), theta_itera(:,2), 'r');