ML Notes: Week 2 - Multivariate Linear Regression

1. The basic theory of the multivariate linear regression

Hypothesis: $h_{\theta }(x)={\theta }_0{x}_0+{\theta }_1{x}_1+\dots +{\theta }_n{x}_n={\theta }^{T}X$

Parameters: ${\theta }_0,{\theta }_1,\dots ,{\theta }_n$

Cost Function: $J({\theta }_0,{\theta }_1,\dots ,{\theta }_n)=\frac{1}{2m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)}{)}^2$

We also can use the gradient descent methop to come up with the optimzed $\theta$.

2. Feature scaling

• Method1: $\frac{x_i}{\max -\min }$
• Method2(Mean Normalization): $\frac{x_i-\mu }{\max -\min }$

The data could be scaled which ranges in $-1\le x_i\le 1$, or in $-0.5\le x_i\le 0.5$

3. Learning rate

• Too small: slow convergence
• Too Large: (a) × convergence; (b) × decreas on every iteration; © slow convergence

TRY！！！
$\alpha =0.0001,0.01,0.1,1$

4. Normal equation

We can utilize the equation to solve out the $\theta$ directly.
$$\theta =(X^{T}X{)}^{-1}{X}^{T}y$$

Derivation of the formula:
Cost Function: $J(\theta )=\frac{1}{2m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)}{)}^2$
so, we can vectorization the Cost Function as follows:
$$\begin{array}{rl}J(\theta )& =\frac{1}{2}\underset{1\ast m}{\underbrace{(X\theta -y{)}^T}}\underset{m\ast 1}{\underbrace{(X\theta -y)}}\\ & =\frac{1}{2}({\theta }^T{X}^{T}X\theta -{\theta }^T{X}^{T}y-y^{T}X\theta -y^{T}y）\end{array}$$
*the $m$ could be igonred.


The $\theta$ that fit to $\frac{\partial J(\theta )}{\partial \theta }=0$ could be considered as the optimum, so
$$\begin{array}{rl}\frac{\partial J(\theta )}{\partial \theta }& =\frac{1}{2}(2X^{T}X\theta -X^{T}y-(y^{T}X{)}^T-0)\\ & =\frac{1}{2}(2X^{T}X\theta -X^{T}y-X^{T}y-0)\\ & =X^{T}X\theta -X^{T}y=0\end{array}$$
$$X^{T}X\theta =X^{T}y$$
we can solve out that $\underset{n\ast 1}{\theta }=(\underset{n\ast m}{X^T}\underset{m\ast n}{X}{)}^{-1}\underset{n\ast m}{X^T}\underset{m\ast 1}{y}$

*(1)$\frac{\partial A\theta }{\partial \theta }=A^T$

*(2)$\frac{\partial {\theta }^{T}A\theta }{\partial \theta }=2A\theta$

%% ============= normal equation ==========
theta_normal = zeros(2,1);
theta_normal = inv(X'*X) * X' * y;

More information: Derivation of the Normal Equation for linear regression

5. Vectorization in univariate gradient descent

• Vectorization

% Vectorization to calculate the \theta
itera = 3000;
theta_matrix = [0 0];
theta_itera = zeros(itera,2); % record all the theta values during the process
for j = 1:itera
    theta_itera(j,:) = theta_matrix;
    hypothesis = X * theta_matrix';
    theta_matrix = theta_matrix - (alpha/m) * ((hypothesis - y)'* X);
end

• “for” Loop

% "for" loop to calculate the \theta
itera = 3000;
theta_itera = zeros(length(y),2);
for j = 1:itera
    theta_itera(j,:) = theta';  % record all the theta values during the process
    hypothesis = X * theta;
    for i = 1:theta_length
        theta(i) = theta(i) - (alpha/m) * ((hypothesis - y)'* X(:,i));  
    end

end

**** What if $X^{T}X$ is non-invertible?

(1) Delete the linearly dependent features (e.g. $x2=2x1$);
(2) Delete some features to make m(# sample) $\le$ n(# features);
(3) Use regularization.