Linear Ragression with multiple variables

Linear Ragression with multiple variables

---

Multiple features

​ Hypothesis:

$$\begin{aligned} h_{\theta}(x) &= \theta^{T} \times X \\ \vec{\theta} &= [\theta_{0}, \theta_{1},\theta_{2} \cdots,\theta_{n}]\\ \vec{X} &= [1,x_1,x_2 \cdots,x_n] \end{aligned}$$

---

Gradient descent for multiple variables

​ Hypothesis:

$$h_{\theta} (x) = \theta^{T} \times X$$
​ Parameters: $\vec{\theta} = [\theta_{0}, \theta_{1},\theta_{2},\cdots, \theta_{n}]$

​ Cost function:

$$\begin{aligned} J(\theta_{0},\theta_{1},\theta_{2},\cdots,\theta_{n}) &= \frac{1}{2m}\sum_{i=1}^{m} (h_{\theta}(x^{(i)}) - y^{(i)})^2\\ J(\vec{\theta}) &= \frac{1}{2m}\sum_{i=1}^{m} (h_{\theta}(x^{(i)}) - y^{(i)})^2\\ \theta_{j} &:= \theta_{j} - \alpha\frac{\partial}{\partial\theta_{j}}J(\vec{\theta})\\ \theta_{j} &:= \theta_{j} - \alpha \frac{1}{m}\sum_{i=1}^{m}(h_{\theta}(x_j^{(i)}) - y_j^{(i)})\times x_j^{(i)}\\ (simultaneously \quad& update \quad \theta_{j} \quad for \quad j=0,1,2,\cdots,n) \end{aligned}$$

---

Practice I : Feature Scaling

​ Feature Scaling

​ E.g. $x_1 = size(0 - 2000 feet^2) ==> x_1 = \frac{size(feet^2)}{2000}$

​ $x_2 = number \quad of \quad bedrooms(1-5) ==> x_2 = \frac{number\_of\_bedrooms}{5}$

​ Mean normalization

​ E.g. $x_1 = \frac{size(feet^2) - 1000}{2000}$

---

Practice II : Learning rate

​ $J(\theta)$ should decrease after every iteration.

​

​ Gradient descent not working, Use smaller $\alpha$.

​ Summary

​ - if $\alpha$ is too small : slow convergence.

​ - if $\alpha$ is too large : $J(\theta)$ may not decrease on every iteration; may not converge.

---

Features and polynomial regression

​ Feature scaling !!

---

Normal equation

​ Normal equation: Method to solve for $\theta$ analytically.

​ Intuition：

​ $\theta \in \mathbb{R}$ $J(\theta_{0}, \theta_{1}, \theta_{2},\cdots, \theta_{n}) = \frac{1}{2m} \sum_{i=1}^{m} (h_{\theta}(x^{(i)}) - y^{(i)})^2$

​ $\frac{\partial}{\partial\theta_j} J(\theta) = \cdots = 0$ (for every j)

​ Solve for $\theta_{0}, \theta_{1}, \theta_{2},\cdots, \theta_{n}$

​ So： （not use Features scaling!）

​ $\vec{\theta} = (\theta_0, \theta_1,\cdots,\theta_n)$ $X = (\vec{x^{(1)^{T}}},\vec{x^{(2)^{T}}},\vec{x^{(3)^{T}}},\cdots,\vec{x^{(m)^{T}}})$

​ $\vec{x^{(i)^{T}}} = (x_{0}^{i},x_{1}^{i},x_{2}^{i},\cdots,x_{n}^{i})$ $\vec{y^{T}} = (y^{(1)}, y^{(2)}, \cdots, y^{(m)})$

​ $\theta = (X^T X)^{-1}X^Ty$

​ Octave: pinv(x'*x)*x'*y

​ $m$ training examples, $n$ features

Gradient Descent	Normal Equation
Need to choose $\alpha$	No need to choose $\alpha$
Needs many iterations	Dont’ need to iterate
Work well even when n is large	Need to compute $(X^TX)^{-1}$
	Slow if n is very large ($O(n^{3})$)

Tips

​ 我们有如下定义：

​ 对于一个函数 $f: \mathbb{R}^{m\times n} -> \mathbb{R}$，将$m \times n$ 的矩阵$A$映射为一个实数，定义 $f$ 为对A 的偏导 为：

$$\nabla_{A} f (A) = \left[ \matrix{ \frac{\partial f}{\partial A_{11}} & \cdots & \frac{\partial f}{\partial A_{1n}}\\ \vdots & \ddots & \vdots\\ \frac{\partial f}{\partial A_{n1}} & \cdots & \frac{\partial f}{\partial A_{nn}} } \right]$$
​ 引入矩阵的 迹 ： $tr$ 。 对于一个 $n$ 阶方阵 $A$ , $tr (A) = \sum_{i=1} ^{n} A_{ii}$

​ 容易得出如下性质：

$$\begin{align} tr(AB) &= tr(BA) \\ tr(A) &= tr(A^T) \\ tr(A + B) &= tr(A) + tr(B) \\ tr(\alpha A) &= \alpha tr (A) \\ \nabla_Atr(AB) &= B^T\\ \nabla_{A^T}f (A) &= (\nabla f(A))^T\\ \nabla_A tr(ABA^TC) &= CAB + C^TAB^T\\ \nabla_A |A| &= |A|(A^{-1})^T \end{align}$$
​ 对于一个向量 $z$ , 有 $z^T z = \sum_{i=1}^{n} z_i^2$ 则：
$$\begin{aligned} \frac{1}{2} (X\theta - \vec{y})^T(X\theta - \vec{y}) &= \frac{1}{2}\sum_{i=1}^{m}(h_{\theta}(x^{(i)}) - y^{(i)})^2 \\ & = J(\theta) \end{aligned}$$
​ 最小化 $J(\theta)$ ，就要求解它关于 $\theta$ 的导数：
$$\begin{aligned} \nabla_{\theta} J(\theta) &= \nabla_{\theta} \frac{1}{2}(X\theta - \vec{y})^T(X\theta - \vec{y})\\ &= \frac{1}{2} \nabla_{\theta}(\theta^TX^TX\theta - \theta^TX^T\vec{y}-{\vec{y}}^TX\theta+{\vec{y}}^T\vec{y})\\ &= \frac{1}{2} \nabla_{\theta}tr(\theta^TX^TX\theta - \theta^TX^T\vec{y}-{\vec{y}}^TX\theta+{\vec{y}}^T\vec{y}) & (tr(\alpha) = \alpha) \\ &= \frac{1}{2} \nabla_{\theta}(tr(\theta^TX^TX\theta) - 2tr({\vec{y}}^TX\theta))& (tr(A) =tr(A^T))\\ &= \frac{1}{2}(X^TX\theta + X^TX\theta - 2X^T\vec{y})&(\nabla_Atr(AB) = B^T)\\ &= X^TX\theta - X^T\vec{y}\\ &= 0 \end{aligned}$$
得出正规方程 ：
$$\begin{aligned} X^TX\theta &= X^T\vec{y}\\ \theta &= (X^TX)^{-1}X^T\vec{y} \end{aligned}$$
​ Octave 的 pinv 在 $(X^TX)$ 不可逆的情况下依旧可以得出 $\theta$ 的解。

​