Linear Regression

Notation

m = Number of training examples
x = input variable/features
y = output variable/ target variable
(x,y) = one training example
(x(i),y(i)) = ith training example(右上角)

Hypothesis

Cost Function（又称平方误差函数）

cost function中出现1/2的原因：因为有平方，与后面梯度算法进行求导时多出来的2相互抵消。

Gradient descent

• 梯度下降概括：

• 梯度下降算法：

α:学习速率，用来控制梯度下降时，迈出多大的步子
随着不断下降，倒数项越来越小，在达到最优解时为0，此时θ将不会再发生改变。

• 导数项推导

Gradient descent for linear regression

“Batch” Gradient descent

Each step of gradient descent uses all the training examples.（如上）

Programming Exercise

城市人数预测食品利益（one variable）
训练集（城市人数，利益）

1. 训练集可视化

data = load('ex1data1.txt');
X = data(:, 1); y = data(:, 2);
m = length(y);
plotData(X, y);

function plotData(x, y)
plot(x,y,'rx','MarkerSize',10);
ylabel('Profit in $10,000s');
xlabel('Population of City in 10,000s');
end

2. costFunction

X = [ones(m, 1), data(:,1)];         %在左侧添加一列，元素为1
theta = zeros(2, 1);
iterations = 1500;
alpha = 0.01;
J = computeCost(X, y, theta);

function J = computeCost(X, y, theta)
m = length(y);
h = X*theta;
J = sum((h - y).^2) / (2*m);
end

3. Gradient Descent

theta = gradientDescent(X, y, theta, alpha, iterations);

function [theta, J_history] = gradientDescent(X, y, theta, alpha, num_iters)
m = length(y); % number of training examples
J_history = zeros(num_iters, 1);
for iter = 1:num_iters
	h = X * theta;
	%theta(1,1) = theta(1,1) - (alpha / m) * sum(h - y);
 	%theta(2,1) = theta(2,1) - (alpha / m) * (X(:,2)')*(h - y);
 	theta = theta - (alpha / m) * X'*(h - y);         %尽量向量化
 	J_history(iter) = computeCost(X, y, theta);
end
end

4. 预测函数

plot(X(:,2), X*theta, '-')

5. 可视化代价函数J

theta0_vals = linspace(-10, 10, 100);
theta1_vals = linspace(-1, 4, 100);
J_vals = zeros(length(theta0_vals), length(theta1_vals));
for i = 1:length(theta0_vals)
    for j = 1:length(theta1_vals)
         t = [theta0_vals(i); theta1_vals(j)];
         J_vals(i,j) = computeCost(X, y, t);
    end
end
% Because of the way meshgrids work in the surf command, we need to  transpose J_vals before calling surf, or else the axes will be flipped
%由于meshgrids在surf命令中的工作方式，我们需要在调用surf之前转换J值，否则轴将翻转。
J_vals = J_vals';
% Surface plot
figure;
surf(theta0_vals, theta1_vals, J_vals)
xlabel('\theta_0'); ylabel('\theta_1');

% Contour plot
figure;
% Plot J_vals as 15 contours spaced logarithmically between 0.01 and 100
contour(theta0_vals, theta1_vals, J_vals, logspace(-2, 3, 20))
xlabel('\theta_0'); ylabel('\theta_1');
hold on;
plot(theta(1), theta(2), 'rx', 'MarkerSize', 10, 'LineWidth', 2);